- 5263 Views
- 1 replies
- 0 kudos
unpersist doesn't clear
from pyspark.sql import SparkSessionfrom pyspark import SparkContext, SparkConffrom pyspark.storagelevel import StorageLevelspark = SparkSession.builder.appName('TEST').config('spark.ui.port','4098').enableHiveSupport().getOrCreate()df4 = spark.sql('...
- 5263 Views
- 1 replies
- 0 kudos
- 0 kudos
Thank you so much for taking time and explaining the concepts
- 0 kudos
- 3624 Views
- 2 replies
- 1 kudos
Bug report: the delimiter option does not work when run on DLT
I have a semicolon separated file in an ADLS container that's been added to Unity Catalog as an External location.When I run the following code on an all-purpose cluster, it runs ok and displays the schema.import dlt @dlt.table def test_data_csv(): ...
- 3624 Views
- 2 replies
- 1 kudos
- 1 kudos
@Retired_mod can you confirm that .option("delimiter", ";") is ignored when run in a DLT pipeline? (please see the post above) My colleage confirmed the behavior.
- 1 kudos
- 1018 Views
- 0 replies
- 0 kudos
Question for exam project
For my exam i have to do a small project for the company im interning at. I am creating a datawarehouse where i will have to transfer data from another database, and then transforming it to a star schema. would databricks be good for this, or is it t...
- 1018 Views
- 0 replies
- 0 kudos
- 5504 Views
- 1 replies
- 1 kudos
Get exceptionTraceId details
I'm getting the following error: module.consumer_stage_catalog.databricks_external_location.catalog: Creating... ╷ │ Error: cannot create external location: AWS IAM role does not have READ permissions on url s3://[bucket name]/catalogs. Please conta...
- 5504 Views
- 1 replies
- 1 kudos
- 4597 Views
- 1 replies
- 0 kudos
Resolved! Can we pass parameters thru SQL UDF's?
Is it possible to pass a parameter to a SQL UDF to another SQL UDF that is called by the first SQL UDF?Below is an example where I would like to call tbl_filter() from tbl_func() by passing the tbl_func.a_val parameter to tbl_filter(). Obviously, I c...
- 4597 Views
- 1 replies
- 0 kudos
- 2211 Views
- 0 replies
- 0 kudos
company profile
At Inspired Elements, we redefine living spaces in London, offering bespoke fitted wardrobes and fitted kitchens that seamlessly blend functionality with exquisite design. Our commitment to innovation and quality ensures every piece is a work of art,...
- 2211 Views
- 0 replies
- 0 kudos
- 7313 Views
- 1 replies
- 1 kudos
Resolved! Αdd columns delta table
Hello. Do you know if you can add columns at a specific position (before / after a column) by altering a delta table ?
- 7313 Views
- 1 replies
- 1 kudos
- 1 kudos
yes, using the FIRST or AFTER parameter.https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-alter-table-manage-column.html#add-column
- 1 kudos
- 9033 Views
- 1 replies
- 0 kudos
Pivot on multiple columns
I want to pass multiple column as argument to pivot a dataframe in pyspark pivot likemydf.groupBy("id").pivot("day","city").agg(F.sum("price").alias("price"),F.sum("units").alias("units")).show(). One way I found is to create multiple df with differ...
- 9033 Views
- 1 replies
- 0 kudos
- 7745 Views
- 4 replies
- 4 kudos
Unity catalog - external table lastUpdateversion
We are currently upgrading our Lakehouse to use the Unity Catalog benefits. We will mostly use external tables because alle our DETLA tables are already stored in Azure Storage. I try to figure out how to update the table property "delta.lastUpdateve...
- 7745 Views
- 4 replies
- 4 kudos
- 4 kudos
I am in the same boat.That is the reason I opted to use managed tables instead. OK; it means migrating tables and changing notebooks but besides not having to struggle with external tables, you also get something in return (liquid clustering f.e.).
- 4 kudos
- 1214 Views
- 2 replies
- 0 kudos
checklist for : process to move and deploy in the prod
Hi Team,Could you please help me with best practices to move and deploy (code, workspace, notebooks, etc) in the prod?Regards,Phanindra
- 1214 Views
- 2 replies
- 0 kudos
- 0 kudos
the most important is to use Repos!Link your workspace with git and use feature branches and pull requests to promote code/notebooks.Check the databricks docs on Repos. If you have further questions; shoot.
- 0 kudos
- 2746 Views
- 1 replies
- 0 kudos
Archival Strategy for Delta tables
Hi Team, We would like to define the archival strategy for data. Could you please share best practices /guide me on the below are the 3 use cases Case-1: On-Prem SQL and Oracle Data which is more than 20 years and they wanted to bring them into clou...
- 2746 Views
- 1 replies
- 0 kudos
- 0 kudos
case 1: I'd extract the data from the db to a data lake (cold storage if that is possible, that is cheaper) using an ETL tool like Data Factory, Glue etc. Then the archiving can take place. Perhaps also create a backup of the data on a 2nd data lak...
- 0 kudos
- 4241 Views
- 2 replies
- 0 kudos
Databricks setup/deployment checklist/best practices
Hi Team, could you please share or guide us on any checklist/best practices for Databricks setup/deployment?
- 4241 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Phani1 , here are some best practices https://github.com/Azure/AzureDatabricksBestPractices/tree/master and you could take these points as your "checklist".Choose the right Databricks Workspace:Decide on the appropriate Azure region for your Data...
- 0 kudos
- 3121 Views
- 1 replies
- 0 kudos
Resolved! Read file with Delta Live Tables from external location (Unity Catalog)
As far as I understand, Delta Live Tables should now support reading data from an external location, but I can’t get it to work. I’ve added an ADLS container to Unity Catalog as an external location. There’s a folder in the container containing an ex...
- 3121 Views
- 1 replies
- 0 kudos
- 0 kudos
I misspelled the folder name; I got it working now The error message could have been more informative
- 0 kudos
- 1401 Views
- 0 replies
- 0 kudos
Highly Performant Data Ingestion and Processing Pipelines
Hi everyone,I am working on a project that requires highly performant pipelines for managing data ingestion, validation, and processing large data volumes from IOT devices.I am interested in knowing:- The best way to ingest from EventHub/Kafka sinks-...
- 1401 Views
- 0 replies
- 0 kudos
- 3150 Views
- 2 replies
- 1 kudos
Databricks Certified Associate Developer for Apache Spark 3.0 - Python sample paper
I am trying to get sample papers for Databricks Certified Associate Developer for Apache Spark 3.0 - Python before I attempt my exam
- 3150 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi, there, This exam has a Practice Exam, I think that's what you are looking for.
- 1 kudos
