Databases are the most common source for companies, organizations, institutions, and government agencies to be publishing APIs from, making their most valuable digital assets available for internal, partner, and public consumption across the web, mobile, network, voice, and device applications they depend on. Making databases a primary target for us here at Streamdata.io. We depend on simple web APIs that are deployed from common database sources, to deliver our event-driven architecture.
To help us better understand the landscape, we are targeting over 40 separate database platforms for review. Working to understand what the API deployment opportunities are around each platform, and help understand where the streaming possibilities reside. Here are the 40+ database solutions we have on our platform currently:
– Airtable (website) (@twitter) (@github) – Airtable works like a spreadsheet, but gives you the power of a database to organize anything.
– Apollo (website) (@twitter) (@github) – Apollo is an incrementally-adoptable data stack that manages the flow of data between clients and backends. Based on GraphQL, it gives you a principled, unified, and scalable API for developing modern apps on top of services.
– ArangoDB (website) (@twitter) (@github) – ArangoDB is an easy to use mostly memory, high performance, open source NoSQL database with a unique combination of features like graph & geo algorithms.
– AWS Database Migration Service (website) – AWS Database Migration Service helps you migrate databases to AWS easily and securely. The source database remains fully operational during the migration, minimizing downtime to applications that rely on the database. The AWS Database Migration Service can migrate your data to and from the most widely used commercial and open-source databases. The service supports homogenous migrations such as Oracle to Oracle, as well as heterogeneous migrations between different database platforms, such as Oracle to Amazon Aurora or Microsoft SQL Server to MySQL.
– AWS DynamoDB (website) – Amazon DynamoDB is a fast and flexible NoSQL database service for all applications that need consistent, single-digit millisecond latency at any scale. It is a fully managed cloud database and supports both document and key-value store models. Its flexible data model and reliable performance make it a great fit for mobile, web, gaming, ad tech, IoT, and many other applications. Start today by downloading the local version of DynamoDB, then read our Getting Started Guide.
– AWS Elastic MapReduce (website) – Amazon EMR provides a managed Hadoop framework that makes it easy, fast, and cost-effective to process vast amounts of data across dynamically Github scalable Amazon EC2 instances. You can also run other popular distributed frameworks such as Apache Spark, HBase, Presto, and Flink in Amazon EMR, and interact with data in other AWS data stores such as Amazon S3 and Amazon DynamoDB.Amazon EMR securely and reliably handles a broad set of big data use cases, including log analysis, web indexing, data transformations (ETL), machine learning, financial analysis, scientific simulation, and bioinformatics.
– AWS Kinesis Analytics (website) – Amazon Kinesis Analytics is the easiest way to process streaming data in real-time with standard SQL without having to learn new programming languages or processing frameworks. Amazon Kinesis Analytics enables you to create and run SQL queries on streaming data so that you can gain actionable insights and respond to your business and customer needs promptly.Amazon Kinesis Analytics takes care of everything required to run your queries continuously and scales automatically to match the volume and throughput rate of your incoming data. With Amazon Kinesis Analytics, you only pay for the resources your queries consume. There is no minimum fee or setup cost.
– AWS Kinesis Firehose (website) – Amazon Kinesis Firehose is the easiest way to load streaming data into AWS. It can capture, transform and load streaming data into Amazon Kinesis Analytics, Amazon S3,Amazon Redshift, and Amazon Elasticsearch Service, enabling near real-time analytics with existing business intelligence tools and dashboards you’re already using today. It is a fully managed service that automatically scales to match the throughput of your data and requires no ongoing administration. It can also batch, compress and encrypt the data before loading it, minimizing the amount of storage used at the destination and increasing security.
– AWS RDS (website) – Amazon Relational Database Service (Amazon RDS) makes it easy to set up, operate, and scale a relational database in the cloud. It provides cost-efficient and resizable capacity while managing time-consuming database administration tasks, freeing you up to focus on your applications and business. Amazon RDS provides you six familiar database engines to choose from, including Amazon Aurora,PostgreSQL, MySQL, Oracle, and Microsoft SQL Server.
– AWS Redshift (website) – Amazon Redshift is a fast, fully managed, petabyte-scaledata warehouse that makes it simple and cost-effective to analyze all your data using your existing business intelligence tools. Start small for $0.25 per hour with no commitments and scale to petabytes for $1,000 per terabyte per year, less than a tenth the cost of traditional solutions. Customers typically see 3x compression, reducing their costs to $333 per uncompressed terabyte per year.
– AWS Server Migration Service (website) – AWS Server Migration Service (SMS) is an agentless service that makes it easier and faster for you to migrate thousands of on-premises workloads to AWS. AWS SMS allows you to automate, schedule, and track incremental replications of live server volumes, making it easier for you to coordinate large-scale server migrations.
– AWS Snowball (website) – Snowball is a petabyte-scale data transport solution that uses secure appliances to transfer large amounts of data into and out of the AWS cloud. Using Snowball addresses common challenges with large-scale data transfers, including high network costs, long transfer times, and security concerns. Transferring data with Snowball is simple, fast, secure, and can be as little as one-fifth the cost of high-speed Internet.With Snowball, you don’t need to write any code or purchase any hardware to transfer your data.
– AWS Storage Gateway Service (website) – The AWS Storage Gateway service seamlessly enables hybrid storage between on-premises storage environments and the AWS Cloud. It combines a multi-protocol storage appliance with highly efficient network connectivity to Amazon cloud storage services, delivering local performance with virtually unlimited scale. Customers use it in remote offices and data centers for hybrid cloud workloads, backup and restore, archive, disaster recovery, and tiered storage.The Storage Gateway virtual appliance connects seamlessly to your local infrastructure as a file server, as a volume, or as a virtual tape library (VTL). This seamless connection makes it simple for organizations to augment existing on-premises storage investments with the high scalability, extreme durability and low cost of cloud storage.
– Azure Data Lake Store (website) – The Data Lake store provides a single repository where you can capture data of any size type and speed simply without forcing changes to your application as the data scales. In the store, data can be shared for collaboration with enterprise-grade security. It is also designed for high-performance processing and analytics from HDFS applications (i.e. Azure HDInsight, Data Lake analytics service, Hortonworks, Cloudera, MapR) and tools, including support for low latency workloads. For example, data can be ingested in real-time from sensors and devices for IoT solutions, or from online shopping websites into the store without the restriction of fixed limits on account or file size, unlike current offerings in the market.
– Azure DocumentDB (website) – Azure DocumentDB is a fully managed NoSQL document database service that offers querying and transaction-processing over schema-free data, predictable and reliable performance, and rapid development.
– Azure SQL Database (website) – Make building and maintaining applications easier and more productive. With built-in intelligence that learns app patterns and adapts to maximize performance, reliability and data protection, SQL Database is a cloud database built for developers.
– Compose (website) (@twitter) (@github) – Build Your Best with MongoDB and Elasticsearch. Compose is a fully managed platform used by developers to deploy, host and scale databases. Compose deployments scale seamlessly as your data grows. Spend more time on your data model, less time capacity planning. This is how everyone should run databases. Backups, replication, complete monitoring. Three essentials of a production database deployment. Spend your time building apps, let us worry about production operations.
– Crate.io (website) (@twitter) (@github) – An open source real-time SQL database with integrated search that is simple to scale. It is ideal for storing and analyzing IoT and machine data in real-time.
– Datical (website) (@twitter) – The database release velocity gap is slowing application release. Get started today & release applications faster with Daticals database release automation tools!
– DBmaestro (website) (@twitter) (@github) – We provide Database Enforced Change Management solution that enables you full control over database development processes and enhance database DevOps performance.
– Dydra.com (website) (@twitter) (@github) – Cloud-hosted RDF & SPARQL database service by @datagraph. Request an invite to the beta program at http://t.co/Sat0oPE8fu
– Google Biquery (website) – BigQuery is Googles fully managed, petabyte scale, low cost enterprise data warehouse for analytics. BigQuery is serverless. There is no infrastructure to manage and you dont need a database administrator, so you can focus on analyzing data to find meaningful insights using familiar SQL. BigQuery is a powerful Big Data analytics platform used by all types of organizations, from startups to Fortune 500 companies.
– Google Cloud Dataproc (website) – Use Google Cloud Dataproc, an Apache Hadoop, Apache Spark, Apache Pig, and Apache Hive service, to process easily big datasets at a low cost. Control your costs by quickly creating managed clusters of any size and turning them off when you’re done. Cloud Dataproc integrates across Google Cloud Platform products, giving you a powerful and complete data processing platform.
– Google Cloud Datastore (website) – Cloud Datastore is a highly-scalable NoSQL database for your applications. Cloud Datastore automatically handles sharding and replication, providing you with a highly available and durable database that scales automatically to handle your application load. Cloud Datastore provides a myriad of capabilities such as ACID transactions, SQL-like queries, indexes and much more.
– Google Cloud Spanner (website) – Cloud Spanner is the first and only relational database service that is both strongly consistent and horizontally scalable. With Cloud Spanner, you enjoy all the traditional benefits of a relational database: ACID transactions, relational schemas (and schema changes without downtime), SQL queries, high performance, and high availability. But unlike any other relational database service, Cloud Spanner scales horizontally, to hundreds or thousands of servers, so it can handle the highest of transactional workloads. With automatic scaling, synchronous data replication, and node redundancy, Cloud Spanner delivers up to 99.999% (five 9s) of availability for your mission-critical applications. In fact, Google’s internal Spanner service has been handling millions of queries per second from many Google services for years.
– Google Cloud SQL (website) – Cloud SQL is a fully-managed database service that makes it easy to set up, maintain, manage, and administer your relational PostgreSQL BETA and MySQL databases in the cloud. Cloud SQL offers high performance, scalability, and convenience. Hosted on Google Cloud Platform, Cloud SQL provides a database infrastructure for applications running anywhere.
– Google Fusion Tables (website) – Fusion Tables is an experimental data visualization web application to gather, visualize, and share data tables.
– Heroku (website) (@twitter) (@github) – Heroku is a platform as a service (PaaS) that enables developers to build, run, and operate applications entirely in the cloud.
– MarkLogic (website) (@twitter) (@github) – MarkLogic provides enterprise solutions for your business by enabling operational big data applications with NoSQL databases and data integration.
– Neo4j (website) (@twitter) (@github) – Meet Neo4j: The graph database platform powering todays mission-critical enterprise applications, including artificial intelligence, fraud detection and recommendations.
– OpsGen (@twitter) – There’s a better way to build connectivity for high-value enterprise, mobile, IoT, and startup applications. OpsGen can help you easily and rapidly create, manage, and scale your API. Visit our website and find your way!
– Periscope Data (website) (@twitter) (@github) – Periscope Data brings all your data together in a single platform and delivers SQL, Python, and R in one solution. Easily warehouse, analyze, visualize and share data from across your business.
– Ragic (website) (@twitter) (@github) – Design powerful Web databases for your business with simple Excel-like interface. Business data management solution that just works.
– Reindex (website) (@twitter) (@github) – Building apps and tools for GraphQL.
– Salesforce (website) (@twitter) (@github) – Build more meaningful and lasting relationships and connect with your customers across sales, customer service, marketing, communities, apps, analytics, and more using our Customer Success Platform. Try for Free.
– Scaphold (@twitter) (@github) – A real-time GraphQL backend as a service. Build web, mobile, & IOT apps faster with React, Angular, Relay, and Apollo using our hosted data and integrations platform. Beats Firebase & Parse.
– SecureDB (@twitter) – SecureDB is the fastest way to encrypt and store the data in the cloud. SecureDB comes built-in with the following modules: 1) Encrypted User Profiles 2) Encrypted Custom Data 3) Encrypted Files. Traditionally, implementing encryption is a complicated, expensive and time-consuming exercise. SecureDB makes implementing strong encryption simple, fast and inexpensive.
– SlashDB (website) (@twitter) (@github) – SlashDB is an API service provider that focuses primarily on delivering APIs (Application programming interface) from your database. Once it is installed on any web server, it will connect your internal databases and construct a REST/HTTP web service, …
– Snowflake Software (website) (@twitter) – Making Aviation Data Accessible and Easy to Use. The Laminar Data Platform manages flight, weather and aeronautical data to unlock operational value.
– Visual Paradigm (website) (@twitter) – Visual Paradigm, a software design tool tailored for agile software projects. It supports UML, BPMN, ERD, DFD, SysML. It also supports use cases, wireframeing, code engineering, etc.
We are looking to write more stories about the API deployment opportunities available across the database platform universe. To do this, we need to better understand the API deployment capabilities of each of these database solutions. Looking at their native services, but also open-sourceAPI, and external connectors that will get the job done. The easier it is to deploy a web API from any of these databases, the easier it will be to stream data using our proxy.
If there is a database platform you depend on not listed here, please let us know. We want to make sure this list is as complete as possible, and we know we’ve missed a number of platforms. Amazon is where we are working at the moment, moving on to Google and Azure platforms next, then we’ll move to some of the smaller providers, exploring the opportunities for partnership, as well as the delivery of plugins and connections. Until we’ve mapped out the database landscape, and developed a healthy understanding where the potential for streaming data resides.
Photo Credit: dirkcuys
Follow us on social