- 4480 Views
- 2 replies
- 2 kudos
Bug report: the delimiter option does not work when run on DLT
I have a semicolon separated file in an ADLS container that's been added to Unity Catalog as an External location.When I run the following code on an all-purpose cluster, it runs ok and displays the schema.import dlt @dlt.table def test_data_csv(): ...
- 4480 Views
- 2 replies
- 2 kudos
- 2 kudos
@Retired_mod can you confirm that .option("delimiter", ";") is ignored when run in a DLT pipeline? (please see the post above) My colleage confirmed the behavior.
- 2 kudos
- 1191 Views
- 0 replies
- 0 kudos
Question for exam project
For my exam i have to do a small project for the company im interning at. I am creating a datawarehouse where i will have to transfer data from another database, and then transforming it to a star schema. would databricks be good for this, or is it t...
- 1191 Views
- 0 replies
- 0 kudos
- 6382 Views
- 1 replies
- 2 kudos
Get exceptionTraceId details
I'm getting the following error: module.consumer_stage_catalog.databricks_external_location.catalog: Creating... ╷ │ Error: cannot create external location: AWS IAM role does not have READ permissions on url s3://[bucket name]/catalogs. Please conta...
- 6382 Views
- 1 replies
- 2 kudos
- 5856 Views
- 1 replies
- 0 kudos
Resolved! Can we pass parameters thru SQL UDF's?
Is it possible to pass a parameter to a SQL UDF to another SQL UDF that is called by the first SQL UDF?Below is an example where I would like to call tbl_filter() from tbl_func() by passing the tbl_func.a_val parameter to tbl_filter(). Obviously, I c...
- 5856 Views
- 1 replies
- 0 kudos
- 2643 Views
- 0 replies
- 0 kudos
company profile
At Inspired Elements, we redefine living spaces in London, offering bespoke fitted wardrobes and fitted kitchens that seamlessly blend functionality with exquisite design. Our commitment to innovation and quality ensures every piece is a work of art,...
- 2643 Views
- 0 replies
- 0 kudos
- 8600 Views
- 1 replies
- 1 kudos
Resolved! Αdd columns delta table
Hello. Do you know if you can add columns at a specific position (before / after a column) by altering a delta table ?
- 8600 Views
- 1 replies
- 1 kudos
- 1 kudos
yes, using the FIRST or AFTER parameter.https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-alter-table-manage-column.html#add-column
- 1 kudos
- 10073 Views
- 1 replies
- 0 kudos
Pivot on multiple columns
I want to pass multiple column as argument to pivot a dataframe in pyspark pivot likemydf.groupBy("id").pivot("day","city").agg(F.sum("price").alias("price"),F.sum("units").alias("units")).show(). One way I found is to create multiple df with differ...
- 10073 Views
- 1 replies
- 0 kudos
- 9487 Views
- 4 replies
- 4 kudos
Unity catalog - external table lastUpdateversion
We are currently upgrading our Lakehouse to use the Unity Catalog benefits. We will mostly use external tables because alle our DETLA tables are already stored in Azure Storage. I try to figure out how to update the table property "delta.lastUpdateve...
- 9487 Views
- 4 replies
- 4 kudos
- 4 kudos
I am in the same boat.That is the reason I opted to use managed tables instead. OK; it means migrating tables and changing notebooks but besides not having to struggle with external tables, you also get something in return (liquid clustering f.e.).
- 4 kudos
- 1622 Views
- 2 replies
- 0 kudos
checklist for : process to move and deploy in the prod
Hi Team,Could you please help me with best practices to move and deploy (code, workspace, notebooks, etc) in the prod?Regards,Phanindra
- 1622 Views
- 2 replies
- 0 kudos
- 0 kudos
the most important is to use Repos!Link your workspace with git and use feature branches and pull requests to promote code/notebooks.Check the databricks docs on Repos. If you have further questions; shoot.
- 0 kudos
- 6049 Views
- 2 replies
- 0 kudos
Databricks setup/deployment checklist/best practices
Hi Team, could you please share or guide us on any checklist/best practices for Databricks setup/deployment?
- 6049 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Phani1 , here are some best practices https://github.com/Azure/AzureDatabricksBestPractices/tree/master and you could take these points as your "checklist".Choose the right Databricks Workspace:Decide on the appropriate Azure region for your Data...
- 0 kudos
- 3780 Views
- 1 replies
- 0 kudos
Resolved! Read file with Delta Live Tables from external location (Unity Catalog)
As far as I understand, Delta Live Tables should now support reading data from an external location, but I can’t get it to work. I’ve added an ADLS container to Unity Catalog as an external location. There’s a folder in the container containing an ex...
- 3780 Views
- 1 replies
- 0 kudos
- 0 kudos
I misspelled the folder name; I got it working now The error message could have been more informative
- 0 kudos
- 1642 Views
- 0 replies
- 0 kudos
Highly Performant Data Ingestion and Processing Pipelines
Hi everyone,I am working on a project that requires highly performant pipelines for managing data ingestion, validation, and processing large data volumes from IOT devices.I am interested in knowing:- The best way to ingest from EventHub/Kafka sinks-...
- 1642 Views
- 0 replies
- 0 kudos
- 3830 Views
- 2 replies
- 1 kudos
Databricks Certified Associate Developer for Apache Spark 3.0 - Python sample paper
I am trying to get sample papers for Databricks Certified Associate Developer for Apache Spark 3.0 - Python before I attempt my exam
- 3830 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi, there, This exam has a Practice Exam, I think that's what you are looking for.
- 1 kudos
- 1442 Views
- 1 replies
- 0 kudos
Infrastructure reverse engineering
How can we do infrastructure reverse engineering in Databricks? Any kind of solution accelerator or tool that can scan through the environment and identify the solutions /components that are used in the Databricks environment and the complexity withi...
- 1442 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Team,Could you please help us on this.Regards,Phanindra
- 0 kudos
- 2251 Views
- 1 replies
- 0 kudos
Got Suspended while taking Databricks Certified Data Analyst Associate Assessment
Hi Team, What I experienced today from the proctor was so not nice, my experience with proctor today was very frustrating and pathetic, I was taking my assessment today 11-26-2023, 2pm, I was already on between question 22nd to 25th when my assessmen...
- 2251 Views
- 1 replies
- 0 kudos
- 0 kudos
@osalawu Sorry to hear you had an issue with your exam. In order to protect your Webassessor account information, please file a ticket with our support team. Please include your Webassessor login ID, the exam, and a couple of dates and times that wil...
- 0 kudos
