- 8009 Views
- 4 replies
- 8 kudos
Resolved! spark managed tables
Hi, I recently started learning about spark. I was studying about spark managed tables. so as per docs " spark manages the both the data and metadata". Assume that i have a csv file in s3 and I read it into data frame like below.df = spark.read .for...
- 8009 Views
- 4 replies
- 8 kudos
- 8 kudos
Yes, @Raviiit DBFS (Databricks File System) is a distributed file system used by Databricks clusters. DBFS is an abstraction layer over cloud storage (e.g. S3 or Azure Blob Store), allowing external storage buckets to be mounted as paths in the DBFS ...
- 8 kudos
- 9900 Views
- 5 replies
- 0 kudos
File copy in adls
i am using dbutils.fs.copy(abfss://container/provsn/filen[ame.txt,abfss://container/data/sasam.txt)while.trying this copy method to copy the files it is showing urisyntax exception near the square bracket how can i read and copy it
- 9900 Views
- 5 replies
- 0 kudos
- 0 kudos
From looking at stack trace, it looks like URIException. Easiest solution would be renaming the file so that there are no square brackets in the filename. If this is not an option, it might help to URLEncode the path - https://stackoverflow.com/que...
- 0 kudos
- 7341 Views
- 3 replies
- 2 kudos
Passing values between notebook tasks in Workflow Jobs
I have created a Databricks workflow job with notebooks as individual tasks sequentially linked. I assign a value to a variable in one notebook task (ex: batchid = int(time.time()). Now, I want to pass this batchid variable to next notebook task.What...
- 7341 Views
- 3 replies
- 2 kudos
- 2 kudos
@brickster You would use dbutils.jobs.taskValues.set() and dbutils.jobs.taskValues.get().See docs for more details: https://docs.databricks.com/workflows/jobs/share-task-context.html
- 2 kudos
- 12386 Views
- 6 replies
- 1 kudos
Resolved! On-Premise SQL Server Ingestion to Databricks Bronze Layer
Hello everyone!So I want to ingest tables with schemas from the on-premise SQL server to Databricks Bronze layer with Delta Live Table and I want to do it using Azure Data Factory and I want the load to be a Snapshot batch load, not an incremental lo...
- 12386 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi @Parsa Bahraminejad Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best an...
- 1 kudos
- 4472 Views
- 2 replies
- 0 kudos
Creating new group
Can someone help me by providing steps for creating a new group, as I could not able to find it anywhere? Actually, I wanted to create a new group for Hyderabad, India which I could not able to find in the Groups sections.@Kaniz Fatma @Sujitha Ramam...
- 4472 Views
- 2 replies
- 0 kudos
- 8901 Views
- 4 replies
- 3 kudos
Resolved! I have a question about the VACUUM feature!
History is piled up as aboveFor testing, I want to erase the history of the table with the VACUUM command."set spark.databricks.delta.retentionDurationCheck.After the option "enabled = False" was given, the command "VACUUM del_park retain 0 hours;" w...
- 8901 Views
- 4 replies
- 3 kudos
- 3 kudos
Executing VACUUM performs garbage cleanup on the table directory. By default, a retention threshold of 7 days will be enforced. Please follow the below steps to perform VACCUM: 1.) SET spark.databricks.delta.retentionDurationCheck.enabled false; This...
- 3 kudos
- 1597 Views
- 0 replies
- 0 kudos
SparkException: Job aborted due to stage failure when attempting to run grid_pointascellid
I am attempting to apply Mosaic's `grid_pointascellid` method on a spark dataframe with `lat`, `lon` columns.```import pyspark.sql.functions as F# Create a Spark DataFrame with a lat and lon columndf = spark.createDataFrame([("point1", 10.0, 20.0),("...
- 1597 Views
- 0 replies
- 0 kudos
- 1388 Views
- 0 replies
- 0 kudos
Mosaic's grid_boundary method returns inconsistent geometries
I am applying mosaic's `grid_boundary` method on a spark DataFrame containing a set of `h3_hex_ids`. The geometries returned are not consistent. i.e they could be either `lat, long` or `long, lat`.Here's a sample data```import pyspark.sql.functions a...
- 1388 Views
- 0 replies
- 0 kudos
- 10114 Views
- 2 replies
- 3 kudos
Resolved! Delta Log checkpoints not being created?
It is mentioned in the delta protocol that checkpoints for delta tables are created every 10 commits - however when I modify a table after >10 separate operations (producing >10 separate json files in the _delta_log directory), no checkpoint files ar...
- 10114 Views
- 2 replies
- 3 kudos
- 3 kudos
As the latest update now checkpointing of delta tables are created for every 100 commits. This is done for some improvement purpose.If you want to have a checkpoint file for delta table for every 10 commits or after any desired commits. You can cust...
- 3 kudos
- 9153 Views
- 5 replies
- 3 kudos
Resolved! Issue with Apache Spark™ Programming with Databricks course
Hello,I found an issue with the Apache Spark™ Programming with Databricks courses on Databricks Academy when trying to do the labs. The mount that the courses use for training data is failing with what looks to me like an authentication issue (see sc...
- 9153 Views
- 5 replies
- 3 kudos
- 3 kudos
I found the course Git Repo at (https://github.com/databricks-academy/apache-spark-programming-with-databricks-english), this works so using that instead of the 'apache-spark-programming-with-databricks.dbc' file available in the learning portal. #DA...
- 3 kudos
- 1840 Views
- 0 replies
- 0 kudos
SparkSession configuration at the Job level is not getting applied
I am trying to get the spark default metrics from the application to statsd sink at Job level not cluster level. So I configured the necessary configuration in the Spark context and spark session in code. And in a local system, which means a single n...
- 1840 Views
- 0 replies
- 0 kudos
- 1560 Views
- 0 replies
- 0 kudos
R Notebooks start before all libraries have been installed
Has anyone else come across an issue where their R notebook executes cells before all libraries in the cluster have been installed?These are libraries in the cluster configuration since the start (not ones I've just put in). Please see attached my tw...
- 1560 Views
- 0 replies
- 0 kudos
- 1774 Views
- 0 replies
- 1 kudos
DataBricks Certification Exam Got Suspended. Require support for the same.
Hi,I applied for Databricks Certified: Data Engineer Professional certification on 5th July 2023. The test was going fine for me but suddenly there was an alert from the system (I think I was in proper angle in front of camera and was genuinely givin...
- 1774 Views
- 0 replies
- 1 kudos
- 725 Views
- 0 replies
- 0 kudos
DataBricks Certification Exam Got Suspended. Require support for it.
Hi,I applied for Databricks Certified: Data Engineer Professional certification on 5th July 2023. The test was going fine for me but suddenly there was an alert from the system (I think I was in proper angle in front of camera and was genuinely givin...
- 725 Views
- 0 replies
- 0 kudos
- 1715 Views
- 0 replies
- 0 kudos
load data from sql server : python works well but spark does not
Hi, i have a problem with my on-premise sql conection from databricks.i have my python code wich use pymssql and it is working well but my spark code doest not and i am using the same credential for both, my spark code is """# Read data from SQL Serv...
- 1715 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
