/
Many organizations use NetSuite as their core ERP system and Snowflake as their primary data warehouse. As data volumes grow and business processes become more interconnected, the question of how to keep these systems in sync becomes more important.
Traditional integrations often move data in one direction only, from NetSuite to Snowflake. However, modern business requirements frequently call for information to flow both ways. This is where the concept of bidirectional, or "two way sync," comes into play.
Two way sync allows updates made in either NetSuite or Snowflake to be reflected in the other system, supporting a range of operational and analytical use cases. This article reviews the technical architecture and practical considerations for implementing such a solution.
Bidirectional sync is a process where data changes in both NetSuite and Snowflake are detected and synchronized so that each system contains the same, up-to-date information. This approach ensures that business operations and analytics remain aligned.
Organizations often require this functionality to enable real-time updates of NetSuite records based on analytics insights generated in Snowflake. For example, sales or inventory data analyzed in Snowflake can trigger automated adjustments in NetSuite, eliminating manual entry and reducing errors.
Maintaining data consistency across ERP and data warehouse environments helps prevent discrepancies, supports compliance, and enhances the accuracy of reporting. Two way sync also supports workflows where operational teams in NetSuite require immediate access to insights or actions derived from data models and analyses performed in Snowflake.
Traditional NetSuite to Snowflake integration is typically one-way and uses ETL (Extract, Transform, Load) pipelines. These pipelines move data out of NetSuite and load it into Snowflake, usually on a scheduled basis. Data flows in a single direction, and any change in NetSuite is eventually reflected in Snowflake, but not the other way around.
Bidirectional or two way sync operates differently. Changes in either NetSuite or Snowflake can trigger updates in the other system, so both databases are kept in alignment. This means that updates, deletions, or new records in Snowflake can be pushed back to NetSuite, and vice versa.
This approach introduces complexity that does not exist with one-way ETL pipelines. When both systems can change the same data, conflicts can occur. For example, if a customer record is updated in NetSuite and Snowflake at the same time, the system determines which change to keep.
A bidirectional sync between NetSuite and Snowflake involves a series of connected components that move, transform, and validate data as it travels in both directions. The architecture consists of several core pieces, each responsible for a stage in the sync process.
Event producers detect when records change in either NetSuite or Snowflake. In NetSuite, triggers such as user actions, scheduled scripts, or webhooks can capture these changes. Webhooks send notifications in real time when records are created, updated, or deleted. In Snowflake, tasks and change data capture processes such as streams monitor and log table changes.
A message queue or change log acts as temporary storage for detected changes before they move to the next step. The message queue receives events from both NetSuite and Snowflake, holding them until each can be processed. This approach ensures that no data is lost if either system or a downstream component is unavailable.
The transformation layer converts data formats as records move between NetSuite objects and Snowflake tables. Field mapping aligns fields from one system to another, while data type conversions change information into the format required by the receiving system. For example, date fields might use different formats in NetSuite and Snowflake, so the transformation layer adjusts these values accordingly.
Synchronizing data between NetSuite and Snowflake in a production environment involves strict security requirements. Enterprise-grade security practices protect sensitive business data and prevent unauthorized access during sync operations.
Modern integrations use OAuth and token-based authentication for both NetSuite and Snowflake APIs. OAuth allows users to grant limited access to applications without sharing their passwords. Token rotation means that authentication tokens are regularly replaced to reduce the risk of unauthorized use.
Network isolation separates integration traffic from the public internet by using private connections, such as VPNs or dedicated tunnels. IP allow listing restricts access to specific IP addresses, so only trusted systems can communicate with NetSuite and Snowflake.
Role-based access control (RBAC) limits permissions within both NetSuite and Snowflake. Each user or integration account is assigned only the access required for its specific tasks, such as reading or writing certain records. RBAC helps prevent unauthorized data access or accidental changes by ensuring that only approved operations are allowed for each role.
Change data capture (CDC) refers to the process of identifying and capturing changes made to data in a system so that these changes can be processed or synchronized elsewhere. For a two way sync between NetSuite and Snowflake, both systems require CDC techniques to detect new, updated, or deleted records.
NetSuite offers APIs and features that help track changes in its records. SuiteTalk, NetSuite's API framework, enables external systems to fetch or interact with NetSuite data. It supports scheduled queries and polling to detect record modifications. SuiteAnalytics Connect provides a way to run saved searches that filter records based on criteria such as last modified date.
Snowflake uses native features called streams and tasks to capture changes in its tables. A stream in Snowflake keeps track of inserts, updates, and deletes that have occurred since the last time the stream was read. Tasks can be set up to execute SQL statements or other processes automatically, allowing for continuous or scheduled processing of these changes.
When real-time change detection is unavailable or interrupted, incremental polling serves as a fallback method. This process involves systems querying NetSuite or Snowflake for records that have changed since a specific timestamp. The system compares the last known update time with the current state to identify new or modified records.
Aligning NetSuite data with Snowflake tables involves converting NetSuite's object-based model into a format that fits Snowflake's relational structure. Mapping strategies focus on linking specific NetSuite records to Snowflake tables, preserving data relationships, and ensuring technical compatibility between fields.
NetSuite uses objects to represent business transactions. Each object type has its own structure and fields, which are mapped to corresponding tables and columns in Snowflake:
NetSuite environments often include custom records and user-defined fields that reflect unique business processes. When mapping these to Snowflake, new tables or columns are created to accommodate the custom data. Dynamic field mapping allows the schema in Snowflake to evolve as new fields are added in NetSuite.
Data type compatibility is important when moving data between NetSuite and Snowflake. Differences in how each system represents dates, numbers, and other data types are addressed during sync operations. Date fields from NetSuite are converted to the appropriate date or timestamp format in Snowflake. Number fields, especially those representing currency, require attention to decimal precision and rounding rules.
Bidirectional sync involves two systems updating and sharing data with each other. Sometimes, problems like data mismatches, failed updates, or changes in data structure can occur. Managing these issues involves clear strategies for detecting, correcting, and preventing inconsistencies.
When both NetSuite and Snowflake change the same record at the same time, a conflict occurs. To resolve this, each update is marked with a timestamp that shows when it happened. Systems then compare timestamps to decide which change is newer. Some sync setups also use a source of truth rule, which means one system is chosen as the final authority for certain data types or fields.
Network problems, system outages, or temporary errors can cause sync operations to fail. To handle these issues, systems use retry and replay queues. Failed sync events are placed in a queue and automatically retried later. If an event cannot be processed after several attempts, it moves to a dead letter queue for manual review.
Sometimes, NetSuite or Snowflake changes the structure of their data by adding new columns or fields. Schema versioning tracks these changes and organizes sync operations so that both systems stay compatible. When a schema update occurs, the sync process uses version numbers to manage the transition.
Managing the performance and cost of a two way sync between NetSuite and Snowflake involves balancing data movement speed, system constraints, and financial planning. Key considerations include API usage, compute resource allocation, and the choice between real-time and batch processing.
NetSuite enforces governance limits on its APIs, which restrict the number of operations that can occur within a set period. Exceeding these limits results in delayed or failed API calls. Sync architectures use request batching, which groups multiple data changes into a single API call to reduce the total number of requests.
Snowflake uses virtual warehouses for processing data, and each warehouse can be sized to match the sync workload. Choosing an appropriate warehouse size involves considering the number of records, frequency of updates, and expected data processing times. Auto-scaling enables Snowflake to adjust compute resources automatically as workload demands increase or decrease.
Data syncs can occur in real time or in batches. Real-time sync transfers changes as they happen, offering low latency but potentially increasing API usage and compute costs. Batch sync processes collect changes over a set period and transfer them together, which can be more efficient but introduces latency.
Organizations decide between building a custom sync solution, adopting an integration platform (iPaaS), or using Stacksync by evaluating technical resources, operational requirements, and long-term costs. Each approach has unique implications for implementation, maintainability, and compliance.
Building a custom bidirectional sync requires software engineers with expertise in NetSuite and Snowflake APIs, data modeling, and system integration. Ongoing support includes bug fixes, updates for API changes, and monitoring for failures. iPaaS solutions provide low-code tools but may still require technical configuration and regular oversight.
Stacksync is designed for fast deployment and low maintenance, relying on no-code connectors and built-in monitoring. Small teams can configure and manage syncs without deep integration expertise.
Total cost of ownership includes licensing, infrastructure, and personnel expenses. Custom code incurs significant upfront development costs, plus ongoing expenses for cloud infrastructure, monitoring tools, and engineering resources. iPaaS platforms charge subscription fees, with additional costs for connectors, data volume, and premium support.
Custom-built solutions require dedicated effort to document security controls, pass compliance audits, and maintain certifications such as SOC 2, ISO 27001, or GDPR. iPaaS providers may offer compliance features, but customers are typically responsible for configuring data protection and audit trails. Stacksync maintains enterprise-grade compliance, including SOC 2 Type II, ISO 27001, GDPR, and HIPAA.
Implementing two way sync between NetSuite and Snowflake involves a series of architectural choices. Each component in the architecture has a role in ensuring data moves accurately and securely between both systems. Handling change data capture, field mapping, schema evolution, and conflict resolution requires careful planning and clear definitions for system behavior.
Security for production environments relies on enterprise authentication, network configuration, and role-based access controls. Sync performance is managed by understanding API rate limits, right-sizing compute resources, and balancing batch versus real-time operations based on data volume and latency tolerance.
Stacksync supports organizations by providing a platform designed for secure, real-time, bi-directional synchronization between systems such as NetSuite and Snowflake. For a technical architecture discussion or to review specific requirements, schedule a session at https://cal.com/rubenburdin/stacksync-demo.
Transaction logs and backup snapshots are created before each sync operation. If a sync fails, the system restores data from the last successful backup and replays only the transactions that have been validated.
Testing schema changes involves creating a separate environment that uses a copy of production data. All mapping rules are validated in this environment, and parallel sync processes are run to compare outcomes with the current production system.
Field-level permissions can be set up in the sync tool to control which custom fields are shared. Filtered views are used to ensure only approved data elements are exposed to Snowflake.
Licensing costs for NetSuite are based on API usage and the number of user licenses. Snowflake costs are determined by the amount of data processed and stored, so expenses increase with higher data volumes and more frequent processing.
