- 10565 Views
- 7 replies
- 8 kudos
How to get data scraped from the web into your data storage
I learning data bricks for the first time following the book that is copywrited in 2020 so I imagine it might be a little outdated at this point. What I am trying to do is move data from an online source (in this specific case using shell script but ...
- 10565 Views
- 7 replies
- 8 kudos
- 8 kudos
In Databricks, you can install external libraries by going to the Clusters tab, selecting your cluster, and then adding the Maven coordinates for Deequ. This represents the best b2b data enrichment services in Databricks.In your notebook or script, y...
- 8 kudos
- 3914 Views
- 2 replies
- 0 kudos
Switching SAS Tokens Mid-Script With Spark Dataframes
Hey all, my team has settled on using directory-scoped SAS tokens to provision access to data in our Azure Gen2 Datalakes. However, we have encountered an issue when switching from a first SAS token (which is used to read a first parquet table in the...
- 3914 Views
- 2 replies
- 0 kudos
- 8584 Views
- 5 replies
- 2 kudos
Resolved! [13.3] Vacuum on table fails if shallow clone without write access exists
Hello everyone,We use unity catalog, separating our dev, test and prod data into individual catalogs.We run weekly vacuums on our prod catalog using a service principal that only has (read+write) access to this production catalog, but no access to ou...
- 8584 Views
- 5 replies
- 2 kudos
- 2 kudos
Are you using Unity Catalog in single user access mode? If yes, could you try using shared access mode.
- 2 kudos
- 4139 Views
- 2 replies
- 1 kudos
Dashboard update through API
Hi,I would like to create / update dashboard definition based on the json file. How can one do it? I tried the following:databricks api post /api/2.0/preview/sql/dashboards/$dashboard_id --json @file.json But it does not update the widgets...How can...
- 4139 Views
- 2 replies
- 1 kudos
- 1 kudos
To programmatically create/update dashboards in Databricks using a JSON file, you can use the Databricks REST API's workspace/export and workspace/import endpoints. Generate a JSON representation of your dashboard using workspace/export, modify it as...
- 1 kudos
- 5146 Views
- 3 replies
- 1 kudos
structured streaming schema inference
I want to stream data from kinesis using DLT. the Data is in json format. How can I use structured streaming to automatically infer the schema? I know auto-loader has this feature but it doesn't make sense for me to use autoloader since my data is st...
- 5146 Views
- 3 replies
- 1 kudos
- 1 kudos
I wanted to use Databricks for this. I don't want to depend on AWS Glue. Same way I could do it with AutoLoader...
- 1 kudos
- 3226 Views
- 3 replies
- 1 kudos
Resolved! cluster metrics databricks runtime 13.1
hello everyone, how to collect metrics provided by clusters metrics databricks runtime 13.1 using bash script
- 3226 Views
- 3 replies
- 1 kudos
- 1 kudos
hi, @Aubert: Currently, you could only use static downloadable snapshots.https://docs.databricks.com/en/compute/cluster-metrics.html Regards,
- 1 kudos
- 2982 Views
- 1 replies
- 1 kudos
How to write only file on to the Blob or ADLS from Databricks?
Hi All,I am trying to write a csv file on to the blob and ADLS from databricks notebook using pyspark and a separate folder is created with the mentioned filename and a partition is created within the folder.I want only file to be written. Can anyone...
- 2982 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Simha , This is expected behavior. Spark always creates an output directory when writing the data and it divides the result into multiple part files. This is because multiple executors write the result into the output directory. We cannot make th...
- 1 kudos
- 1840 Views
- 0 replies
- 0 kudos
Uncaught Error: Script error for jupyter-widgets/base
I have been using ipywidgets for quite a while in several notebooks in Databricks, but today things have completely stopped working with the following error;Uncaught Error: Script error for "@jupyter-widgets/base" http://requirejs.org/docs/errors.htm...
- 1840 Views
- 0 replies
- 0 kudos
- 31211 Views
- 7 replies
- 1 kudos
Resolved! Writing modular code in Databricks
Hi All, Could you please suggest to me the best way to write PySpark code in Databricks,I don't want to write my code in Databricks notebook but create python files(modular project) in Vscode and call only the primary function in the notebook(the res...
- 31211 Views
- 7 replies
- 1 kudos
- 1 kudos
Certainly! To write PySpark code in Databricks while maintaining a modular project in VSCode, you can organize your PySpark code into Python files in VSCode, with a primary function encapsulating the main logic. Then, upload these files to Databricks...
- 1 kudos
- 40711 Views
- 5 replies
- 1 kudos
Resolved! How can I write a single file to a blob storage using a Python notebook, to a folder with other data?
When I use the following code: df .coalesce(1) .write.format("com.databricks.spark.csv") .option("header", "true") .save("/path/mydata.csv")it writes several files, and when used with .mode("overwrite"), it will overwrite everything in th...
- 40711 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi Daniel,May I know, how did you fix this issue. I am facing similar issue while writing csv/parquet to blob/adls, it creates a separate folder with the filename and creates a partition file within that folder.I need to write just a file on to the b...
- 1 kudos
- 3893 Views
- 1 replies
- 1 kudos
Asset Bundles: Dynamic job cluster insertion in jobs
Hi!As we are migrating from dbx to asset bundles we are running into some problems with the dynamic insertion of job clusters in the job definition as with dbx we did this nicely with jinja and defined all the clusters in one place and a change in th...
- 3893 Views
- 1 replies
- 1 kudos
- 6248 Views
- 2 replies
- 0 kudos
Resolved! Thread leakage when connection cannot be established
During the execution of the next code we can observe a lost thread that will never end:@Testpublic void pureConnectionErrorTest() throws Exception { try { DriverManager.getConnection(DATABRICKS_JDBC_URL, DATABRICKS_USERNAME, DATABRICKS_PASS...
- 6248 Views
- 2 replies
- 0 kudos
- 0 kudos
This issue is reported as fixed since v2.6.34. I validated version 2.6.36- it works normal. Many thanks to the developers for the work done!
- 0 kudos
- 1838 Views
- 1 replies
- 0 kudos
Delta Live Table streaming pipeline
How do I do a simple left join of a static table and a streaming table under catalog in the streaming pipeline of a Delta Live Table?
- 1838 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @rt-slowth I would like to share with you the Databricks documentation, which contains details about stream-static table joins https://docs.databricks.com/en/delta-live-tables/transform.html#stream-static-joins Stream-static joins are a good choic...
- 0 kudos
- 3817 Views
- 2 replies
- 0 kudos
Row-level Concurrency and Liquid Clustering compatibility
The documentation is a little ambiguous:"Row-level concurrency is only supported on tables without partitioning, which includes tables with liquid clustering."https://docs.databricks.com/en/release-notes/runtime/14.2.html Tables with liquid clusterin...
- 3817 Views
- 2 replies
- 0 kudos
- 0 kudos
Cluster-on-write is something being worked on. The limitations at the moment have to do with accommodating streaming workloads.I found the following informative:https://www.youtube.com/watch?v=5t6wX28JC_M
- 0 kudos
- 3311 Views
- 1 replies
- 1 kudos
IDENTIFIER clause
Hi all. Just trying to implement adb sql scripts using identifier clause but I have errors like that using an example: DECLARE mytab = 'tab1'; CREATE TABLE IDENTIFIER(mytab) (c1 INT); [UNSUPPORTED_FEATURE.TEMP_VARIABLE_ON_DBSQL] The feature is not...
- 3311 Views
- 1 replies
- 1 kudos
- 1 kudos
@RobsonNLPT - Engineering is still working on the feature that allow DECLARE statements in DBSQL. This is with a tentative ETA of Feb 20 available on preview channel.
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
