- 1885 Views
- 2 replies
- 0 kudos
Resolved! CICD Databricks
Hi TeamI was wondering if there is a document or step by step process to promote code in CICD across various environments of code repository (GIT/GITHUB/BitBucket/Gitlab) with DBx support? [Without involving code repository merging capability of the ...
- 1885 Views
- 2 replies
- 0 kudos
- 0 kudos
Please refer this related thread on CICD in Databricks https://community.databricks.com/s/question/0D53f00001GHVhMCAX/what-are-some-best-practices-for-cicd
- 0 kudos
- 1922 Views
- 3 replies
- 0 kudos
- 1922 Views
- 3 replies
- 0 kudos
- 4878 Views
- 5 replies
- 4 kudos
Resolved! Auto Loader for Shape File
Hello: As you can see from below link, that it support 7 file formats. I am dealing with GeoSpatial Shape files and I want to know if Auto Loader can support Shape Files ???Any help on this is greatly appreciated. Thanks. https://docs.microsoft.com/...
- 4878 Views
- 5 replies
- 4 kudos
- 4 kudos
You could try to use the binary file type. But the disadvantage of this is that the content of the shape files will be put into a column, that might not be what you want.If you absolutely want to use the autoloader, maybe some thinking outside the b...
- 4 kudos
- 1402 Views
- 1 replies
- 0 kudos
cluster not getting created, timing out
Hello - i've been using the Databricks notebook(for pyspark or scala/spark development), and recently have had issues wherein the cluster creation takes a long time to get created, often timing out. Any ideas on how to resolve this ?
- 1402 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Karankaran.alang,What is the error message you are getting? did you get this error while creating/starting a cluster CE?some times these errors are intermittent and go away after a few re-tries.Thank you
- 0 kudos
- 3759 Views
- 2 replies
- 1 kudos
Resolved! Spark StreamingQuery not processing all data from source directory
Hi,I have setup a streaming process that consumers files from HDFS staging directory and writes into target location. Input directory continuesouly gets files from another process.Lets say file producer produces 5 million records sends it to hdfs sta...
- 3759 Views
- 2 replies
- 1 kudos
- 1 kudos
If it helps , you run try running the Left-Anti join on source and sink to identify missing records and see whether the record is in match with the schema provided or not
- 1 kudos
- 2382 Views
- 1 replies
- 1 kudos
How can I get Databricks notebooks to stop cutting off the explain plans?
(since Spark 3.0)Dataset.queryExecution.debug.toFilewill dump the full plan to a file, without concatenating the output as a fully materialized Java string in memory.
- 2382 Views
- 1 replies
- 1 kudos
- 1 kudos
Notebooks really aren't the best method of viewing large files. Two methods you could employ areSave the file to dbfs and then use databricks CLI to download the fileUse the web terminalIn the web terminal option you can do something like "cat my_lar...
- 1 kudos
- 1233 Views
- 1 replies
- 0 kudos
Auto Loader for the Shape File ?.
Hello, As you can see from below link, that it support 7 file formats. I am dealing with GeoSpatial Shape files and I want to know if Auto Loader can support Shape Files ???Any help on this is greatly appreciated. avro: Avro filebinaryFile: Binary f...
- 1233 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Yuv Saha​ ,Currently, shapefiles are not a supported file-type when using auto-loader. Would you be willing to share more about your use case? I am the Product Manager responsible for Geospatial in Databricks, and I need help from customers like ...
- 0 kudos
- 5400 Views
- 1 replies
- 0 kudos
Delta Lake Python
I have setup a virtual environment inside my existing hadoop cluster. Since the current cluster does not have spark >3 , so i installed delta spark using virtual environment. While trying to access the hdfs which is kerberose one, Getting below error...
- 5400 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Vasanth P​ ,Could you please confirm the DBR version that you are using in the cluster? There are multiple DBR versions that is available with different spark and scala version which can utilised directly in databricks cluster.
- 0 kudos
- 5039 Views
- 1 replies
- 3 kudos
Resolved! How to handle configuration for different environment (e.g. DEV, PROD)?
May I know any suggested way to handle different environment variables for the same code base? For example, the mount point of Data Lake for DEV, UAT, and PROD. Any recommendations or best practices? Moreover, how to handle Azure DevOps?
- 5039 Views
- 1 replies
- 3 kudos
- 3 kudos
@Eric Yeung​ , you can put all your configuration parameters in a file (JSON, CONF, YAML whatever you like) and read that file at the beginning of each program.I like to use the ConfigFactory in Scala for example.You only have to make sure the file c...
- 3 kudos
- 10616 Views
- 4 replies
- 1 kudos
Is it possible to pass configuration to a job on high concurrency cluster?
On a regular cluster, you can use:```spark.sparkContext._jsc.hadoopConfiguration().set(key, value)```These values are then available on the executors using the hadoop configuration. However, on a high concurrency cluster, attempting to do so results ...
- 10616 Views
- 4 replies
- 1 kudos
- 1 kudos
I am not sure why you are getting that error on a high concurrency cluster. As I am able to set the configuration as you show above. Can you try the following code instead? sc._jsc.hadoopConfiguration().set(key, value)
- 1 kudos
- 6230 Views
- 3 replies
- 4 kudos
Resolved! Incremental write
Hi All,I have a daily spark job that reads and joins 3-4 source tables and writes the df in a parquet format. This data frame consists of 100+ columns. As this job run daily, our deduplication logic identifies the latest record from each of source t...
- 6230 Views
- 3 replies
- 4 kudos
- 8824 Views
- 5 replies
- 3 kudos
Resolved! Read just the new file ???
Hi guys,How can I read just the new file in a batch process ?Can you help me ? pleasThank you
- 8824 Views
- 5 replies
- 3 kudos
- 3 kudos
What type of file? Is the file stored in a storage account? Typically, you would read and write data with something like the following code: # read a parquet file df = spark.read.format("parquet").load("/path/to/file") # write the data as a file df...
- 3 kudos
- 2056 Views
- 3 replies
- 2 kudos
Resolved! Current DBR is not yet available to this notebook
Any one has an idea why i am getting this error:"The current DBR is not yet available to this notebook. Give it a second and try again!"
- 2056 Views
- 3 replies
- 2 kudos
- 2 kudos
@Meysam az​ - Thank you for letting us know that the issue has been resolved and for the extra information.
- 2 kudos
- 12007 Views
- 2 replies
- 1 kudos
Parsing deeply nested XML in Databricks
Hi Guys,Can someone point me to libraries to parse XML files in Databricks using Python / Scala.Any link to blog / documentations will be helpful.Looked into https://docs.databricks.com/data/data-sources/xml.html.Want to parse XSD's, seem this is exp...
- 12007 Views
- 2 replies
- 1 kudos
- 1 kudos
@Sreedhar Vengala​ - I heard back from the team. As you noted, the feature is still experimental and not supported at this time.I would like to assure you that the team is aware of this. I have no information about a time frame to make this a support...
- 1 kudos
- 3468 Views
- 2 replies
- 2 kudos
Resolved! Is is possible to Mount multiple ADLS Gen2 Storage paths in single workspace
Hello Experts,We are looking on feasibility of mounting more that one ADLS Gen2 storages on a single workspace of databricks.Best RegardsPraveen
- 3468 Views
- 2 replies
- 2 kudos
- 2 kudos
Yes, its possible, we are doing it. Just mount them to different folders like @Werner Stinckens​ is saying.
- 2 kudos
| User | Count |
|---|---|
| 1611 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
