cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

VJ3
by New Contributor III
  • 3108 Views
  • 1 replies
  • 0 kudos

Azure Databricks Notebook Sharing, Notebook Exporting and Notebook Clipboard copy download

Hello,I would like to know in which scenario Azure Databricks User would be able to download Notebook Command Output if Notebook Result Download is disabled. Do we know if Privilege user would be able to share sensitive information with non-privilege...

  • 3108 Views
  • 1 replies
  • 0 kudos
Latest Reply
VJ3
New Contributor III
  • 0 kudos

Thank you Kaniz. Can we disable the exporting of notebook except Source File? If yes, then how do we achieve is?Also, we do not want to share the notebook which has any kind of notebook results, can we use spark.databricks.query.displayMaxRows and se...

  • 0 kudos
Shree23
by New Contributor III
  • 1301 Views
  • 1 replies
  • 0 kudos

scalar function in databricks

Hi Expert,here is sql server scalar function how to convert in databricks functionSQLCREATE function [dbo].[gettrans](@PickupCompany nvarchar(2),@SupplyCountry int, @TxnSource nvarchar(10),@locId nvarchar(50), @ExternalSiteId nvarchar(50))RETURNS INT...

  • 1301 Views
  • 1 replies
  • 0 kudos
Latest Reply
Shree23
New Contributor III
  • 0 kudos

Suggestion pls

  • 0 kudos
alexiswl
by Contributor
  • 1741 Views
  • 1 replies
  • 0 kudos

Resolved! 'Unity Catalog Volumes is not enabled on this instance' error

Hi all,tl;dr I ran the following on a docker-backed personal compute instance (running 13.3-LTS)```%sqlUSE CATALOG hail;USE SCHEMA volumes_testing;CREATE VOLUME 1kg    COMMENT 'Testing 1000 Genomes volume';```But this gives```ParseException: [UC_VOLU...

  • 1741 Views
  • 1 replies
  • 0 kudos
Latest Reply
alexiswl
Contributor
  • 0 kudos

Resolved with the setting "spark.databricks.unityCatalog.volumes.enabled" = "true"

  • 0 kudos
Databricks24
by New Contributor
  • 1443 Views
  • 0 replies
  • 0 kudos

UserAgentEntry added to JDBC URL but not visible in Audit logs

Hi,As part of Databricks Best Practices, I have added 'UserAgentEntry' to JDBC URL that is being created when we are executing SQL statements through the JDBC driver.Sample url - jdbc:databricks://<host>:443;httpPath=<httpPath>; AuthMech=3;UID=token;...

  • 1443 Views
  • 0 replies
  • 0 kudos
nyck33
by New Contributor II
  • 1472 Views
  • 1 replies
  • 0 kudos

Databricks learning festival, but my trial is over

I just emailed the onboarding-help email account to ask for an extension for 2 weeks as I want to complete the Data Engineer course to prepare for my new position. I have 2 accounts where the trial expired, one community account which cannot be used ...

  • 1472 Views
  • 1 replies
  • 0 kudos
Latest Reply
nyck33
New Contributor II
  • 0 kudos

is what happened when trying to sign up with another email.

  • 0 kudos
DBEnthusiast
by New Contributor III
  • 1593 Views
  • 1 replies
  • 1 kudos

More than expected number of Jobs created in Databricks

Hi Databricks Gurus !I am trying to run a very simple snippet :data_emp=[["1","sarvan","1"],["2","John","2"],["3","Jose","1"]]emp_columns=["EmpId","Name","Dept"]df=spark.createDataFrame(data=data_emp, schema=emp_columns)df.show() --------Based on a g...

  • 1593 Views
  • 1 replies
  • 1 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 1 kudos

This widget could not be displayed.
Hi Databricks Gurus !I am trying to run a very simple snippet :data_emp=[["1","sarvan","1"],["2","John","2"],["3","Jose","1"]]emp_columns=["EmpId","Name","Dept"]df=spark.createDataFrame(data=data_emp, schema=emp_columns)df.show() --------Based on a g...

This widget could not be displayed.
  • 1 kudos
This widget could not be displayed.
hukel
by Contributor
  • 2364 Views
  • 1 replies
  • 0 kudos

Resolved! Convert multiple string fields to int or long during streaming

Source data looks like: { "IntegrityLevel": "16384", "ParentProcessId": "10972929104936", "SourceProcessId": "10972929104936", "SHA256Hash": "a26a1ffb81a61281ffa55cb7778cc3fb0ff981704de49f75f51f18b283fba7a2", "ImageFileName": "\\Device\\Harddisk...

  • 2364 Views
  • 1 replies
  • 0 kudos
Latest Reply
hukel
Contributor
  • 0 kudos

Thanks for confirming that the readStream.withColumn() approach is the best available option.  Unfortunately, this will force me to maintain a separate notebook for each of the event types,  but it does work.   I was hoping to create just one paramet...

  • 0 kudos
rpl
by New Contributor III
  • 2288 Views
  • 2 replies
  • 1 kudos

Bug report: the delimiter option does not work when run on DLT

I have a semicolon separated file in an ADLS container that's been added to Unity Catalog as an External location.When I run the following code on an all-purpose cluster, it runs ok and displays the schema.import dlt @dlt.table def test_data_csv(): ...

  • 2288 Views
  • 2 replies
  • 1 kudos
Latest Reply
rpl
New Contributor III
  • 1 kudos

@Retired_mod can you confirm that .option("delimiter", ";") is ignored when run in a DLT pipeline? (please see the post above) My colleage confirmed the behavior. 

  • 1 kudos
1 More Replies
dvmentalmadess
by Valued Contributor
  • 3587 Views
  • 1 replies
  • 1 kudos

Get exceptionTraceId details

I'm getting the following error:  module.consumer_stage_catalog.databricks_external_location.catalog: Creating... â•· │ Error: cannot create external location: AWS IAM role does not have READ permissions on url s3://[bucket name]/catalogs. Please conta...

  • 3587 Views
  • 1 replies
  • 1 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 1 kudos

This widget could not be displayed.
I'm getting the following error:  module.consumer_stage_catalog.databricks_external_location.catalog: Creating... â•· │ Error: cannot create external location: AWS IAM role does not have READ permissions on url s3://[bucket name]/catalogs. Please conta...

This widget could not be displayed.
  • 1 kudos
This widget could not be displayed.
TrevorC
by New Contributor III
  • 2981 Views
  • 1 replies
  • 0 kudos

Resolved! Can we pass parameters thru SQL UDF's?

Is it possible to pass a parameter to a SQL UDF to another SQL UDF that is called by the first SQL UDF?Below is an example where I would like to call tbl_filter() from tbl_func() by passing the tbl_func.a_val parameter to tbl_filter(). Obviously, I c...

  • 2981 Views
  • 1 replies
  • 0 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Is it possible to pass a parameter to a SQL UDF to another SQL UDF that is called by the first SQL UDF?Below is an example where I would like to call tbl_filter() from tbl_func() by passing the tbl_func.a_val parameter to tbl_filter(). Obviously, I c...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
inspiredelement
by New Contributor
  • 1565 Views
  • 0 replies
  • 0 kudos

company profile

At Inspired Elements, we redefine living spaces in London, offering bespoke fitted wardrobes and fitted kitchens that seamlessly blend functionality with exquisite design. Our commitment to innovation and quality ensures every piece is a work of art,...

  • 1565 Views
  • 0 replies
  • 0 kudos
elgeo
by Valued Contributor II
  • 3442 Views
  • 1 replies
  • 1 kudos

Resolved! Αdd columns delta table

Hello. Do you know if you can add columns at a specific position (before / after a column) by altering a delta table ?

  • 3442 Views
  • 1 replies
  • 1 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 1 kudos

yes, using the FIRST or AFTER parameter.https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-alter-table-manage-column.html#add-column

  • 1 kudos
memo
by New Contributor II
  • 6208 Views
  • 1 replies
  • 0 kudos

Pivot on multiple columns

I want to pass multiple column as argument to pivot a dataframe  in pyspark pivot likemydf.groupBy("id").pivot("day","city").agg(F.sum("price").alias("price"),F.sum("units").alias("units")).show(). One way I found is to create multiple df with differ...

  • 6208 Views
  • 1 replies
  • 0 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
I want to pass multiple column as argument to pivot a dataframe  in pyspark pivot likemydf.groupBy("id").pivot("day","city").agg(F.sum("price").alias("price"),F.sum("units").alias("units")).show(). One way I found is to create multiple df with differ...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
rudyevers
by New Contributor III
  • 5091 Views
  • 4 replies
  • 4 kudos

Unity catalog - external table lastUpdateversion

We are currently upgrading our Lakehouse to use the Unity Catalog benefits. We will mostly use external tables because alle our DETLA tables are already stored in Azure Storage. I try to figure out how to update the table property "delta.lastUpdateve...

  • 5091 Views
  • 4 replies
  • 4 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 4 kudos

I am in the same boat.That is the reason I opted to use managed tables instead.  OK; it means migrating tables and changing notebooks but besides not having to struggle with external tables, you also get something in return (liquid clustering f.e.).

  • 4 kudos
3 More Replies
rpl
by New Contributor III
  • 1893 Views
  • 1 replies
  • 0 kudos

Resolved! Read file with Delta Live Tables from external location (Unity Catalog)

As far as I understand, Delta Live Tables should now support reading data from an external location, but I can’t get it to work. I’ve added an ADLS container to Unity Catalog as an external location. There’s a folder in the container containing an ex...

Community Platform Discussions
Delta Live Tables
Unity Catalog
  • 1893 Views
  • 1 replies
  • 0 kudos
Latest Reply
rpl
New Contributor III
  • 0 kudos

I misspelled the folder name; I got it working now  The error message could have been more informative

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Top Kudoed Authors