Hi, what is the recommended way to read data from delta table and write to ADLS gen2 storage in parquet format. In my case i use a notebook to read data do some processing and write it to storage and update delta table with detail of last written da...
I need to pass data between sql azure databases but the columns of some tables are different, the information must go to that column but with a different name.
I am already storing my data as parquet files and have registered them as a table in Databricks. If I want to convert the table to be a Delta table, do I have to do a full read of the data and rewrite it in the Delta format?
more details and programmatic options can be found in the Porting Guide.
Kinesis streams is the Kinesis streaming service. Select this!Kinesis Firehose reads data from a Kinesis stream and writes it e.g. to S3 or Redshift. or Splunk (more details here)
What is differentiator between delta sharing and other cloud sharing platform.
Also, unlike other servers, Delta Sharing internally uses pre-signed URLs to S3, GCS, or ADSL, so data transfer from a client happens at the bandwidth of the underlying cloud object-store. This way the Delta Sharing server scales extremely well and d...
Hi Databricks Community, I have followed the CLV Databricks accelator (https://databricks.com/notebooks/CLV_Part_1_Customer_Lifetimes.html) to do an initial CLV analysis. Thank you for sharing this with the community. My question is that in the note...
I am running this script in Azure Databricks using spark SQL , getting this error below , I am new to Databricks so wondering any tips on troubleshooting thisError in SQL statement: ParseException:mismatched input 'Service_Date' expecting {'(', 'DESC...
Dear all, The Spark JDBC driver (SparkJDBC42.jar) is unable to capture certain information from the below table structure: 1. table level comment 2. the TBLPROPERTIES key-value pair information 3. PARTITION BY information However, it captures the co...
I'm getting the following error when I'm trying to load a h2o model using mlflow for prediction Error: Error Job with key $03017f00000132d4ffffffff$_990da74b0db027b33cc49d1d90934149 failed with an exception: java.lang.IllegalArgumentException:...
I ran this in Databricks and it worked with no issues. I suggest you make sure your wget path is correct, because the one you posted downloads HTML, not the raw csv. That may cause the problem. %sh wget https://raw.githubusercontent.com/mlflow/mlflo...
I'm trying to create a dashboard in Databricks SQL, parameterized by table name. We have a metadata table which contains the names of all the eligible tables, and we use it to populate a drop-down box for the dashboard. This is a simplified version ...
Not sure whether better do ask this in an Azure or Spark subject, but I thought I might get responses appropriate to our use cases here. We have Azure Databricks set up and working, and not had any problems following along the tutorials, but I don't...
Hey guys, I am looking to create a real-time analytics application and I am pretty new to Data engineering. Any advice here would be appreciated. So I have been l appvalleyooking into spark streaming for my transformation process, so th tutuappe ove...
like in pandas I usually do df['columnname'].unique()
Hi, this worked for me. distinct_ids = [x.id for x in data.select('id').distinct().collect()]
Hi guys, im new using databricks and i have a challenge in my new work. routerlogin I need to access to one the database (the database is on DBFS) result of some ETLS trough any service, can be ODBC or by some API. I need to connect there because I...
Use the Simba ODBC connector: https://docs.databricks.com/integrations/bi/jdbc-odbc-bi.html
I have been trying to create a new cluster to use and multiple attempts have gotten stuck in pending: "Finding instances for new nodes, acquiring more instances if necessary" until they time out. Up to today I have had no problems creating clusters ...
This is typically a cloud provider issue. You can file a support ticket if the issue persists.
| User | Count |
|---|---|
| 1601 | |
| 736 | |
| 343 | |
| 284 | |
| 247 |
