# spring-cloud-dataflow

**Repository Path**: liqy153/spring-cloud-dataflow

## Basic Information

- **Project Name**: spring-cloud-dataflow
- **Description**: A microservices-based Streaming and Batch data processing in Cloud Foundry and Kubernetes
- **Primary Language**: Unknown
- **License**: Apache-2.0
- **Default Branch**: 1.0.x
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2021-12-01
- **Last Updated**: 2021-12-01

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

== Spring Cloud Data Flow image:https://build.spring.io/plugins/servlet/buildStatusImage/SCD-BMASTER[Build Status, link=https://build.spring.io/browse/SCD-BMASTER] image:https://badge.waffle.io/spring-cloud/spring-cloud-dataflow.svg?label=ready&title=Ready[Stories Ready, link=https://waffle.io/spring-cloud/spring-cloud-dataflow] image:https://badge.waffle.io/spring-cloud/spring-cloud-dataflow.svg?label=In%20Progress&title=In%20Progress[Stories In Progress, link=https://waffle.io/spring-cloud/spring-cloud-dataflow]

The Spring Cloud Data Flow project provides orchestration for data microservices, including long lived
https://github.com/spring-cloud/spring-cloud-stream[stream] applications and
short lived https://github.com/spring-cloud/spring-cloud-task[task] applications.

== Components

The https://github.com/spring-cloud/spring-cloud-dataflow/tree/master/spring-cloud-dataflow-core[Core]
domain module includes the concept of a *stream* that is a composition of spring-cloud-stream
modules in a linear pipeline from a *source* to a *sink*, optionally including *processor* module(s)
in between. The domain also includes the concept of a *task*, which may be any process that does
not run indefinitely, including https://github.com/spring-projects/spring-batch[Spring Batch] jobs.

The https://github.com/spring-cloud/spring-cloud-dataflow/tree/master/spring-cloud-dataflow-registry[App Registry]
maintains the set of available apps, and their mappings to URIs.
For example, if relying on Maven coordinates, an app's URI would be of the format:
`maven://<groupId>:<artifactId>:<version>`

The Data Flow Server is a Spring Boot application that provides a common REST API and UI. For each
runtime environment there is a different version of the Data Flow Server that depends upon a
deployer SPI implementation for that environment. The github locations for these Data Flow Servers are:

* https://github.com/spring-cloud/spring-cloud-dataflow/tree/master/spring-cloud-dataflow-server-local[Local] (intended for development use only)
* https://github.com/spring-cloud/spring-cloud-dataflow-server-cloudfoundry[Cloud Foundry]
* https://github.com/spring-cloud/spring-cloud-dataflow-server-yarn[Apache YARN]
* https://github.com/spring-cloud/spring-cloud-dataflow-server-mesos[Apache Mesos]
* https://github.com/spring-cloud/spring-cloud-dataflow-server-kubernetes[Kubernetes]

The deployer SPI mentioned above is defined within the https://github.com/spring-cloud/spring-cloud-deployer[Spring Cloud Deployer]
project. That provides an abstraction layer for deploying the apps of a given stream or task and managing their lifecycle.
The github locations for the corresponding Spring Cloud Deployer SPI implementations are:

* https://github.com/spring-cloud/spring-cloud-deployer/tree/master/spring-cloud-deployer-local[Local]
* https://github.com/spring-cloud/spring-cloud-deployer-cloudfoundry[Cloud Foundry]
* https://github.com/spring-cloud/spring-cloud-deployer-yarn[Apache YARN]
* https://github.com/spring-cloud/spring-cloud-deployer-mesos[Apache Mesos]
* https://github.com/spring-cloud/spring-cloud-deployer-kubernetes[Kubernetes]

The https://github.com/spring-cloud/spring-cloud-dataflow/tree/master/spring-cloud-dataflow-shell[Shell]
connects to the Data Flow Server's REST API and supports a DSL that simplifies the process of
defining a stream or task and managing its lifecycle.

Instructions for running the Data Flow Server for each runtime environment can be found in their respective github repositories.

=== Building

Clone the repo and type 

----
$ ./mvnw clean install 
----

For more information on building, see this https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/appendix-building.adoc[link].

=== Contributing

We love contributions.  Follow this https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-docs/src/main/asciidoc/appendix-contributing.adoc[link] for more information on how to contribute.