- 2214 Views
- 2 replies
- 2 kudos
- 2214 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @S Meghala​ ,Please go through this Github link you will get good amount of data here ,this way you can learn morehttps://github.com/AlexIoannides/pyspark-example-projectPlease select my answer as best answer if your query is fulfilled ThanksAvira...
- 2 kudos
- 5727 Views
- 7 replies
- 3 kudos
What happens if I have both DLTs and normal tables in a single notebook?
I've just learned Delta Live Tables on Databricks Academy and have no environment to try it out.I'm wondering what happens to the pipeline if the notebook consists of both normal tables and DLTs. For exampleTable ADLT A that reads and cleans Table AT...
- 5727 Views
- 7 replies
- 3 kudos
- 3 kudos
hey ,@S L​ According to you , you have normal table table A and DLT table Table B , so it will give thrown an error that your upstream table is not streaming Live table and you need to create streaming live table Table a , if you want to use the ou...
- 3 kudos
- 3404 Views
- 2 replies
- 2 kudos
Subquery does not work in Databricks Community version?
I am testing some SQL code based on the book SQL Cookbook Second Edition, available from https://downloads.yugabyte.com/marketing-assets/O-Reilly-SQL-Cookbook-2nd-Edition-Final.pdfBased on Page 43, I am OK with the left join, as shown here:However, w...
- 3404 Views
- 2 replies
- 2 kudos
- 2 kudos
it must have some github link check there or you cans hare your code and data we can help you
- 2 kudos
- 11530 Views
- 4 replies
- 2 kudos
Resolved! java.lang.NoSuchMethodError after upgrade to Databricks Runtime 11.3 LTS
Hi, I am using Databricks and want to upgrade to Databricks runtime version 11.3 LTS which uses Spark 3.3 now. Current system enviroment:Operating System: Ubuntu 20.04.4 LTSJava: Zulu 8.56.0.21-CA-linux64Python: 3.8.10Delta Lake: 1.1.0Target system ...
- 11530 Views
- 4 replies
- 2 kudos
- 13773 Views
- 5 replies
- 3 kudos
Adding to PYTHONPATH in interactive Notebooks
I'm trying to set PYTHONPATH env variable in the cluster configuration: `PYTHONPATH=/dbfs/user/blah`. But in the driver and executor envs it is probably getting overridden and i don't see it.`%sh echo $PYTHONPATH` outputs:`PYTHONPATH=/databricks/spar...
- 13773 Views
- 5 replies
- 3 kudos
- 3 kudos
Update:At last found a (hacky) solution!in the driver I can dynamically set the sys.path in the workers with:`spark._sc._python_includes.append("/dbfs/user/blah/")`combine that with, in the driver:```%load_ext autoreload%autoreload 2```and setting: `...
- 3 kudos
- 5005 Views
- 2 replies
- 2 kudos
DLT UDF and c#
Hello, can I create spark function in .net and use it in DLT table? I would like to encrypt some data, in documentation scala code is being used as an example, but would it be possible to write decryption/encryption function using C# and use it withi...
- 5005 Views
- 2 replies
- 2 kudos
- 2 kudos
It's not possible. SQL Server 2008 contains SQL CLR runtime that runs .NET languages.
- 2 kudos
- 4880 Views
- 3 replies
- 4 kudos
Clarification on merging multiple notebooks and other
1. Do we have any feature like merge the cells from one or more notebooks into other notebook.2. Do we have any feature like multiple cells from excel is copied it into multiple cells in a notebook . Generally all excel data is copied it into one cel...
- 4880 Views
- 3 replies
- 4 kudos
- 4 kudos
1) We can't merge cells right now2)We don't have this feature as well3) We don't have multiple editing right now4)You will know only if you face an error. A Notification will pop up5)You can"t keep running the execution because the cells can be linke...
- 4 kudos
- 4125 Views
- 6 replies
- 30 kudos
DLT PipeLine Understanding
Hey, guys, I hope you are doing very well today I was going through some databricks documentation and I found dlt documentation but when I am trying to implement it, it is not working very well can anyone can share with me whole code step by step and...
- 4125 Views
- 6 replies
- 30 kudos
- 7271 Views
- 6 replies
- 2 kudos
Resolved! Notebooks cells limit
Dear Team,is there a limit in notebook cells in a single notebook in community edition ?
- 7271 Views
- 6 replies
- 2 kudos
- 5273 Views
- 3 replies
- 2 kudos
What exact difference does Auto Loader make?
New to Databricks and here is one thing that confuses me.Since Spark Streaming is already capable of incremental loading by checkpointing. What difference does it make by enabling Auto Loader?
- 5273 Views
- 3 replies
- 2 kudos
- 2 kudos
Auto Loader provides a Structured Streaming source called cloudFiles. Given an input directory path on the cloud file storage, the cloudFiles source automatically processes new files as they arrive, with the option of also processing existing files i...
- 2 kudos
- 11864 Views
- 4 replies
- 6 kudos
Error handling/exception handling in NOtebook
What is a common practice to to write notebook which includes error handling/exception handling.Is there any example which depicts how notebook should be written to include error handling etc.
- 11864 Views
- 4 replies
- 6 kudos
- 6 kudos
runtime looks for handlers (try-catch) that are registered to handle such exceptions
- 6 kudos
- 14854 Views
- 3 replies
- 25 kudos
Understanding Joins in PySpark/Databricks In PySpark, a `join` operation combines rows from two or more datasets based on a common key. It allows you ...
Understanding Joins in PySpark/DatabricksIn PySpark, a `join` operation combines rows from two or more datasets based on a common key. It allows you to merge data from different sources into a single dataset and potentially perform transformations on...
- 14854 Views
- 3 replies
- 25 kudos
- 1989 Views
- 1 replies
- 4 kudos
Error for sparkdl.xgboost import XgboostRegressor
I get the error :cannot import name 'resnet50' from 'keras.applications' (/local_disk0/.ephemeral_nfs/envs/pythonEnv-a3e7b0cc-064d-4585-abfd-6473ed1c1a5b/lib/python3.8/site-packages/keras/applications/__init__.py) It looks like the Keras.applications...
- 1989 Views
- 1 replies
- 4 kudos
- 4 kudos
try to install these libraries via init script some time this happen due to spark version in databricks , libraries can make conflict with Runtime version
- 4 kudos
- 2712 Views
- 0 replies
- 0 kudos
Getting error [DATATYPE_MISMATCH.BINARY_OP_DIFF_TYPES]
[DATATYPE_MISMATCH.BINARY_OP_DIFF_TYPES] Cannot resolve "(DocDate AND orderedhl)" due to data type mismatch: the left and right operands of the binary operator have incompatible types ("STRING" and "DECIMAL(38,6)").; line 67, pos 066. group by 67. or...
- 2712 Views
- 0 replies
- 0 kudos
- 4286 Views
- 5 replies
- 1 kudos
Resolved! slow running query
Hi All, I would you to get some ideas on how to improve performance on a data frame with around 10M rows. adls- gen2df1 =source1 , format , parquet ( 10 m)df2 =source2 , format , parquet ( 10 m)df = join df1 and df2 type =inner join df.count() is ...
- 4286 Views
- 5 replies
- 1 kudos
- 1 kudos
hey @raghu maremanda​ did you get any answer if yes ,please update here, by that other people can also get the solution
- 1 kudos
| User | Count |
|---|---|
| 1630 | |
| 790 | |
| 512 | |
| 349 | |
| 287 |
