- 8113 Views
- 6 replies
- 2 kudos
Resolved! Notebooks cells limit
Dear Team,is there a limit in notebook cells in a single notebook in community edition ?
- 8113 Views
- 6 replies
- 2 kudos
- 6099 Views
- 3 replies
- 2 kudos
What exact difference does Auto Loader make?
New to Databricks and here is one thing that confuses me.Since Spark Streaming is already capable of incremental loading by checkpointing. What difference does it make by enabling Auto Loader?
- 6099 Views
- 3 replies
- 2 kudos
- 2 kudos
Auto Loader provides a Structured Streaming source called cloudFiles. Given an input directory path on the cloud file storage, the cloudFiles source automatically processes new files as they arrive, with the option of also processing existing files i...
- 2 kudos
- 12375 Views
- 4 replies
- 6 kudos
Error handling/exception handling in NOtebook
What is a common practice to to write notebook which includes error handling/exception handling.Is there any example which depicts how notebook should be written to include error handling etc.
- 12375 Views
- 4 replies
- 6 kudos
- 6 kudos
runtime looks for handlers (try-catch) that are registered to handle such exceptions
- 6 kudos
- 16340 Views
- 3 replies
- 25 kudos
Understanding Joins in PySpark/Databricks In PySpark, a `join` operation combines rows from two or more datasets based on a common key. It allows you ...
Understanding Joins in PySpark/DatabricksIn PySpark, a `join` operation combines rows from two or more datasets based on a common key. It allows you to merge data from different sources into a single dataset and potentially perform transformations on...
- 16340 Views
- 3 replies
- 25 kudos
- 2216 Views
- 1 replies
- 4 kudos
Error for sparkdl.xgboost import XgboostRegressor
I get the error :cannot import name 'resnet50' from 'keras.applications' (/local_disk0/.ephemeral_nfs/envs/pythonEnv-a3e7b0cc-064d-4585-abfd-6473ed1c1a5b/lib/python3.8/site-packages/keras/applications/__init__.py) It looks like the Keras.applications...
- 2216 Views
- 1 replies
- 4 kudos
- 4 kudos
try to install these libraries via init script some time this happen due to spark version in databricks , libraries can make conflict with Runtime version
- 4 kudos
- 2955 Views
- 0 replies
- 0 kudos
Getting error [DATATYPE_MISMATCH.BINARY_OP_DIFF_TYPES]
[DATATYPE_MISMATCH.BINARY_OP_DIFF_TYPES] Cannot resolve "(DocDate AND orderedhl)" due to data type mismatch: the left and right operands of the binary operator have incompatible types ("STRING" and "DECIMAL(38,6)").; line 67, pos 066. group by 67. or...
- 2955 Views
- 0 replies
- 0 kudos
- 4776 Views
- 5 replies
- 1 kudos
Resolved! slow running query
Hi All, I would you to get some ideas on how to improve performance on a data frame with around 10M rows. adls- gen2df1 =source1 , format , parquet ( 10 m)df2 =source2 , format , parquet ( 10 m)df = join df1 and df2 type =inner join df.count() is ...
- 4776 Views
- 5 replies
- 1 kudos
- 1 kudos
hey @raghu maremanda​ did you get any answer if yes ,please update here, by that other people can also get the solution
- 1 kudos
- 44704 Views
- 3 replies
- 1 kudos
How to explode an array column and repack the distinct values into one array in DB SQL?
Hi, I am new to DB SQL. I have a table where the array column (cities) contains multiple arrays and some have multiple duplicate values. I need to unpack the array values into rows so I can list the distinct values. The following query works for this...
- 44704 Views
- 3 replies
- 1 kudos
- 11205 Views
- 6 replies
- 33 kudos
Resolved! Timezone understanding
Today I was working in Timezone kind of data but my Singapore user want to see their time in the Data and USA user want to see their time in the datainstead of both, we all are getting UTC time,how to solve this issuePlease guide Data can be anything...
- 11205 Views
- 6 replies
- 33 kudos
- 33 kudos
I got it guys it was happening due to a library conflict now your answers are really helpful I tried all things
- 33 kudos
- 4204 Views
- 5 replies
- 1 kudos
Resolved! Databricks authentication
Hi there!!we are planning to use databricks -tableau on prem integration for reporting. Data would reside in delta lake and using ta leau-databricks connector, user would be able to generate reports from that data .question is: a private end point wi...
- 4204 Views
- 5 replies
- 1 kudos
- 1 kudos
and make sure that you are going with SPARK SQL connection , else it will always fail
- 1 kudos
- 5273 Views
- 3 replies
- 0 kudos
DBFS File Browser Error RESOURCE_DOES_NOT_EXIST:
Hi,I am new to databricks, and was trying to follow some tutorial to upload a file and move it under some different folder. I used DBFS option.While trying to move/rename the file I am getting below error, can you please help to understand why I am g...
- 5273 Views
- 3 replies
- 0 kudos
- 0 kudos
use these three commands and it will workdbutils.fs.ls('dbfs:/FileStore/vehicle_data.csv')dbutils.fs.ls('/dbfs/FileStore/vehicle_data.csv')dbutils.fs.ls('/dbfs/dbfs/FileStore/vehicle_data.csv')ThanksAviral
- 0 kudos
- 3342 Views
- 3 replies
- 10 kudos
- 3342 Views
- 3 replies
- 10 kudos
- 10 kudos
Hi, @Vidula Khanna​, the new version of Databricks still has only 2.12 scala support.
- 10 kudos
- 10765 Views
- 2 replies
- 0 kudos
How to get the count of dataframe rows when reading through spark.readstream using batch jobs?
I am trying to read messages from kafka topic using spark.readstream, I am using the following code to read it.My CODE:df = spark.readStream .format("kafka") .option("kafka.bootstrap.servers", "192.1xx.1.1xx:9xx") .option("subscr...
- 10765 Views
- 2 replies
- 0 kudos
- 0 kudos
You can try this approach:https://stackoverflow.com/questions/57568038/how-to-see-the-dataframe-in-the-console-equivalent-of-show-for-structured-st/62161733#62161733ReadStream is running a thread in background so there's no easy way like df.show().
- 0 kudos
- 10621 Views
- 7 replies
- 7 kudos
Resolved! Copying delta to Azure SQL DB.
How to copy DELTA to AZURE SQL DB using ADF? Earlier we are using parquet format. Now, We have converted parquet to Delta by using below command: CONVERT TO DELTA parquet.path (Azure Blob Path)
- 10621 Views
- 7 replies
- 7 kudos
- 7 kudos
Hi @Aviral Bhardwaj​ in ADF there is an option of a delta lake file you can directly save your file in delta lake format.
- 7 kudos
- 2848 Views
- 2 replies
- 2 kudos
How can I pull all branches at once in Databricks?
Hi,I have cloned a remote repository into my folder in Repos. The repository has several feature branches.When I want to pull any branch, I select the desired branch in Repos and click on "Pull" button. And then I need to select another branch. Is th...
- 2848 Views
- 2 replies
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 566 | |
| 349 | |
| 287 |
