- 3565 Views
- 1 replies
- 6 kudos
Resolved! Are there data types that are not good in Delta format? Does Delta handle images, audio, and video?
- 3565 Views
- 1 replies
- 6 kudos
- 6 kudos
Hi as it is transaction tables (there are history commits and snapshot). I would not store there images or videos as it can be saved few times and you will have high storage costs, it can also be slow when data is big.I would definitely store images,...
- 6 kudos
- 5325 Views
- 4 replies
- 10 kudos
Resolved! How to make sparklyr extension work with Databricks runtime?
Hello. I'm the current maintainer of sparklyr (a R interface for Apache Spark) and a few sparklyr extensions such as sparklyr.flint.Sparklyr was fortunate to receive some contribution from Databricks folks, which enabled R users to run `spark_connect...
- 5325 Views
- 4 replies
- 10 kudos
- 10 kudos
Yes, as Sebastian said. Also, it would be good to know what the error is here. One possible explanation is that the JARs are not copied to the executor nodes. This would be solved by Sebasitian's suggestion.
- 10 kudos
- 1653 Views
- 1 replies
- 0 kudos
- 1653 Views
- 1 replies
- 0 kudos
- 0 kudos
@Rajeswari Gummadi​ - Is this a duplicate of your other thread? I don't see any content and I want to make sure all your questions are answered.
- 0 kudos
- 4809 Views
- 3 replies
- 17 kudos
Resolved! Azure: Permanently purge cluster logs
Is there any way to purge logs via API instead of clicking daily that option:
- 4809 Views
- 3 replies
- 17 kudos
- 7839 Views
- 2 replies
- 1 kudos
AssertionError: assertion failed: Unable to delete the record but I am able to select it though
Is there any reason this command works well:%sql SELECT * FROM datanase.table WHERE salary > 1000returning 2 rows, while the below:%sql delete FROM datanase.table WHERE salary > 1000ErrorError in SQL statement: AssertionError: assertion failed:...
- 7839 Views
- 2 replies
- 1 kudos
- 1 kudos
DELETE FROM (and similarly UPDAT. aren't supported on the Parquet files - right now on Databricks, it's supported for Delta format. You can convert your parquet files into delta using CONVERT TO DELTA, and then this command will work for you.
- 1 kudos
- 15554 Views
- 4 replies
- 4 kudos
Resolved! Unable to save Spark Dataframe to driver node's local file system as CSV file
Running Azure Databricks Enterprise DBR 8.3 ML running on a single node, with Python notebook. I have 2 small Spark dataframes that I am able source via credential passthrough reading from ADLSgen2 via `abfss://` method and display the full content ...
- 15554 Views
- 4 replies
- 4 kudos
- 4 kudos
Modern Spark operates by a design choice to separate storage and compute. So saving a csv to the river's local disk doesn't make sense for a few reasons:the worker nodes don't have access to the driver's disk. They would need to send the data over to...
- 4 kudos
- 37561 Views
- 7 replies
- 0 kudos
Resolved! How to show an image in a notebook using html?
Hi everyone, I just learning how to personalize the databricks notebooks and would like to show a logo in a cell. I installed the databricks cli and was able to upload the image file to the dbfs: I try to display it like this: displayHTML("<im...
- 37561 Views
- 7 replies
- 0 kudos
- 0 kudos
@Paul Hernandez​ @Sean Owen​ @Navneet Tuteja​ I solved this after I also ran into the same issue where my notebook suddenly wouldn't show an image sitting on the driver in an accessible folder - no matter what I was trying in the notebook the display...
- 0 kudos
- 17221 Views
- 3 replies
- 3 kudos
Resolved! Why doesn't my notebook display widgets when I use 'dbutils' while it is displayed with '%sql CREATE WIDGET'?
The widget is not shown when I use dbutils while it works perfect with sql.For example, %sql CREATE WIDGET TEXT state DEFAULT "CA"This one shows me widget.dbutils.widgets.text("name", "Brickster", "Name") dbutils.widgets.multiselect("colors", "oran...
- 17221 Views
- 3 replies
- 3 kudos
- 3 kudos
Hello, Ryan! For some reason, this problem is solved, and now it is working perfectly! I did nothing new, but it is just working now. Thank you!:)
- 3 kudos
- 9087 Views
- 4 replies
- 4 kudos
Resolved! Databricks writeStream checkpoint
I'm trying to execute this writeStream data_frame.writeStream.format("delta") \ .option("checkpointLocation", checkpoint_path) \ .trigger(processingTime="1 second") \ .option("mergeSchema", "true") \ .o...
- 9087 Views
- 4 replies
- 4 kudos
- 4 kudos
You can remove that folder so it will be recreated automatically.Additionally every new job run should have new (or just empty) checkpoint location.You can add in your code before running streaming:dbutils.fs.rm(checkpoint_path, True)Additionally you...
- 4 kudos
- 16197 Views
- 9 replies
- 10 kudos
Resolved! Unable to write csv files to Azure BLOB using pandas to_csv ()
I am using a Py function to read some data from a GET endpoint and write them as a CSV file to a Azure BLOB location.My GET endpoint takes 2 query parameters,param1 and param2. So initially, I have a dataframe paramDf that has two columns param1 and ...
- 16197 Views
- 9 replies
- 10 kudos
- 10 kudos
@Hubert Dudek​ I cant issue a spark command to executor node, throws up an error ,because foreach distributes the processing.
- 10 kudos
- 7032 Views
- 3 replies
- 7 kudos
Resolved! Run Pyspark job of Python egg package using spark submit on databricks
Error: missing application resource​Getting this error while running job with spark submit.​ I have given following parameters while creating job:--conf spark.yarn.appMasterEnv.PYSAPRK_PYTHON=databricks/path/python3--py-files dbfs/path/to/.egg job_m...
- 7032 Views
- 3 replies
- 7 kudos
- 7 kudos
Hi,We tried a simulate the question on our end and what we did was packaged a module inside a whl file.Now to access the wheel file we created another python file test_whl_locally.py. Inside test_whl_locally.py to access the content of the wheel file...
- 7 kudos
- 6437 Views
- 1 replies
- 5 kudos
Resolved! Get package from Nexus repo.
I want to receive a package from Nexus repo both in notebook and job. If anyone has experience with this, please answer me here!
- 6437 Views
- 1 replies
- 5 kudos
- 5 kudos
Nexus repo for the notebook you can use Notebook-scoped libraries with %pip with Use %pip install with the --index-url option. Secret management is available. See example.from UI it is not supported Cluster libraries
- 5 kudos
- 1362 Views
- 0 replies
- 1 kudos
Hi, could you please help me with my question? i have not get any answers.
Hi,could you please help me with my question? i have not get any answers.
- 1362 Views
- 0 replies
- 1 kudos
- 5903 Views
- 1 replies
- 0 kudos
What is the best way to decode protobuf using pyspark?
I am using spark structured streaming to read a protobuf encoded message from the event hub. We use a lot of Delta tables, but there isn't a simple way to integrate this. We are currently using K-SQL to transform into avro on the fly and then use Dat...
- 5903 Views
- 1 replies
- 0 kudos
- 0 kudos
hi @Will Block​ ,I think there is a related question being asked in the past. I think it was this one I found this library, I hope it helps.
- 0 kudos
- 12387 Views
- 8 replies
- 3 kudos
Resolved! error on connecting to Snowflake
Hi team, I'm getting weird error in one of my jobs when connecting to Snowflake. All my other jobs (I've got plenty) work fine. The current one also works fine when I have only one coding step (except installing needed libraries in my very first step...
- 12387 Views
- 8 replies
- 3 kudos
- 3 kudos
@marchello​ I suggest you contact Snowflake to move forward on this one.
- 3 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
