cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

weilin0323
by New Contributor III
  • 140 Views
  • 2 replies
  • 3 kudos

Resolved! How to Apply Encryption Function to a Specific Column

Hello!I would like to apply a function to encrypt a specific column. The UDF is as follows: DROP FUNCTION IF EXISTS EncryptColumn; CREATE FUNCTION EncryptColumn (key_name STRING, encryptcolumn STRING) RETURN base64(aes_encrypt(encryptcolumn, key_nam...

weilin0323_0-1737526512566.png
  • 140 Views
  • 2 replies
  • 3 kudos
Latest Reply
weilin0323
New Contributor III
  • 3 kudos

Hi @MadhuB,The method you provided is feasible, and I later finded other ways to apply UDF:UPDATE table_name SET column_name = EncryptColumn(key_name, column_name) Thank you!

  • 3 kudos
1 More Replies
johngabbradley
by New Contributor II
  • 107 Views
  • 2 replies
  • 0 kudos

Using spark.read.json with a {} literal in my path

I am pulling data from an S3 bucket using spark.read.json like thiss3_uri = "s3://snowflake-genesys/v2.outbound.campaigns.{id}/2025-01-22/00/"       df = spark.read.json(s3_uri)My s3 url has the {id} in the file path.  I have used r"s3://snowflake-ge...

  • 107 Views
  • 2 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @johngabbradley, Would below approach work for you? s3_uri = "s3://snowflake-genesys/v2.outbound.campaigns.{id}/2025-01-22/00/"files = dbutils.fs.ls(s3_uri)file_paths = [file.path for file in files]df = spark.read.json(file_paths)

  • 0 kudos
1 More Replies
Wallace_Selis
by New Contributor
  • 969 Views
  • 1 replies
  • 0 kudos

HELP

I can't log in. After entering the code received in the email, I remain on this screen  

Wallace_Selis_0-1737574404014.png
  • 969 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

If you try in the incognito mode does it shows same error?  

  • 0 kudos
avesel
by New Contributor
  • 907 Views
  • 1 replies
  • 0 kudos

How to refer to repository directories in Workflows

HiI need to refer to a configuration file which resides in a separate directory than the script. The paths and execution withing a notebook/python files work fine. When script is scheduled and uses code repository the directory names look obfuscated ...

  • 907 Views
  • 1 replies
  • 0 kudos
Latest Reply
lauraxyz
Contributor
  • 0 kudos

can you try with relative path?like you get your current path within test_script.py with command like:cur_path = os.getcwd() then get the path to config.yaml with a relative path like:config_path = os.path.abspath(os.path.join(cur_path, f"../config/c...

  • 0 kudos
sensanjoy
by Contributor
  • 220 Views
  • 6 replies
  • 0 kudos

Performance issue when reading data from a view.

Hi All,We are facing some performance issue and I need your help to know what could be the best approach to follow here.Existing: For each region, we have view(Reg1_View,Reg2_View..) to pull data from table( we don't have direct access to table).And ...

View_Vs_Staging_Table.png
  • 220 Views
  • 6 replies
  • 0 kudos
Latest Reply
SharathAbh93
New Contributor II
  • 0 kudos

Does any table hold data of all region 1. if yes. Get a Materialized view created (replacing all_reg_view)2. i see you already tried creating a staging table replacing the all_reg_view. Try creating cluster key along with partition.Cluster key on the...

  • 0 kudos
5 More Replies
thiagoawstest
by Contributor
  • 387 Views
  • 1 replies
  • 0 kudos

No access to databricks console

 Hello, I have the following situation: when trying to configure SSO, it was enabled to allow login using Microsoft, but the problem is that the sessions expired, and now we cannot access with any email, it says that the account is not enabled.How co...

Data Engineering
AWS
dataengineer
  • 387 Views
  • 1 replies
  • 0 kudos
Latest Reply
Miguel_Suarez
Databricks Employee
  • 0 kudos

Hi @thiagoawstest, Please reach out to your Account Executive or Solutions Architect. The will be able to help you with the issue you're experiencing while trying to log in. Best

  • 0 kudos
databricks98
by New Contributor
  • 403 Views
  • 1 replies
  • 0 kudos

Failed to send request to Azure Databricks Cluster

We have scheduled an ADF (Azure Data Factory) pipeline that contains a Lookup activity, which is responsible for fetching the last ingested date from the Databricks catalog(Hive metastore). I attached screenshot please find https://yourimageshare.com...

  • 403 Views
  • 1 replies
  • 0 kudos
Latest Reply
Miguel_Suarez
Databricks Employee
  • 0 kudos

Hi @databricks98,  It seems like there is some issue connecting to your Azure account. Were there any recent changes to firewalls, permissions, or cluster configurations? Could you please check to make sure that the connection between Databricks and ...

  • 0 kudos
jsaddam28
by New Contributor III
  • 49802 Views
  • 25 replies
  • 16 kudos

How to import local python file in notebook?

for example I have one.py and two.py in databricks and I want to use one of the module from one.py in two.py. Usually I do this in my local machine by import statement like below two.py__ from one import module1 . . . How to do this in databricks???...

  • 49802 Views
  • 25 replies
  • 16 kudos
Latest Reply
PabloCSD
Valued Contributor
  • 16 kudos

This alternative worked for us: https://community.databricks.com/t5/data-engineering/is-it-possible-to-import-functions-from-a-module-in-workspace/td-p/5199

  • 16 kudos
24 More Replies
HoussemBL
by New Contributor III
  • 118 Views
  • 2 replies
  • 0 kudos

External tables in DLT pipelines

Hello community,I have implemented a DLT pipeline.In the "Destination" setting of the pipeline I have specified a unity catalog with target schema of type external referring to an S3 destination.My DLT pipeline works well. Yet, I noticed that all str...

  • 118 Views
  • 2 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hello @HoussemBL, You can use below code example: import dlt @dlt.create_streaming_table(name="your_table_name",path="s3://your-bucket/your-path/",schema="schema-definition")def your_table_function():return ( spark.readStream.format("your_format").op...

  • 0 kudos
1 More Replies
momita_s
by New Contributor II
  • 145 Views
  • 2 replies
  • 0 kudos

Resolved! How can we fetch application id in serverless compute in databricks?

Hi All,How can we fetch application id in serverless compute in databricks? We are working to use serverless compute for some jobs. The issue is we are not able to fetch application id in notebook. Earlier we were using spark.sparkContext.application...

  • 145 Views
  • 2 replies
  • 0 kudos
Latest Reply
Alberto_Umana
Databricks Employee
  • 0 kudos

Hi @momita_s, Thanks for your question. As you mentioned above parameters/commands are not available in serverless, I did not find a way to retrieve the applicationID on serverless, I would check internally but this likely requires a feature request.

  • 0 kudos
1 More Replies
srinivasu
by New Contributor II
  • 236 Views
  • 5 replies
  • 0 kudos

Unable to find course materials for the course: Delivery Specialization: CDW Migration Best Practice

Hi,Tried searching everywhere but unable to find course materails for the course: Delivery Specialization: CDW Migration Best Practice.In the video it says see course materials, but I dont see anything with that name anywhere. Please let me know if s...

srinivasu_0-1736756512399.png
  • 236 Views
  • 5 replies
  • 0 kudos
Latest Reply
srinivasu
New Contributor II
  • 0 kudos

I believe it is closed, I'm unable to check the status of the ticket. It is giving an error: "You currently do not have access to Help Center.Please reach out to your admin or send an email to help@databricks.com"If you are able to check the request ...

  • 0 kudos
4 More Replies
UlrikChristense
by New Contributor II
  • 234 Views
  • 5 replies
  • 0 kudos

Apply-changes-table (SCD2) with huge amounts of `rowIsHidden=True` rows

I have a lot of DLT tables creating using the `apply_changes` function with type 2 history. This functions creates a physical table `__apply_changes_storage_<table_name>` and a view on top of this `<table_name>`. The number of rows the physical table...

  • 234 Views
  • 5 replies
  • 0 kudos
Latest Reply
UlrikChristense
New Contributor II
  • 0 kudos

I'm trying, but doesn't seem to change anything. Setting these table properties - when are the "applied"? When the job is run, or as a background thing?

  • 0 kudos
4 More Replies
MichielPovre
by New Contributor II
  • 125 Views
  • 1 replies
  • 1 kudos

Resolved! Delta Live Tables - use cluster-scoped init scripts

Hi All,According the documentation of delta live tables (https://docs.databricks.com/en/delta-live-tables/external-dependencies.html), one can user either global or cluster scoped init scripts. However, I don't see an option to select init scripts in...

Data Engineering
Delta Live Tables
  • 125 Views
  • 1 replies
  • 1 kudos
Latest Reply
AngadSingh
New Contributor III
  • 1 kudos

Hi, You can do it via a cluster policy. It can be achieved in two steps: create a cluster policy with required attributes.You can provide the init_scripts attribute in the policy. For reference: https://learn.microsoft.com/en-us/azure/databricks/admi...

  • 1 kudos
Meenambigai
by New Contributor
  • 69 Views
  • 1 replies
  • 0 kudos

Link for webinar Get Started with Databricks for Data Engineering session

where to find Link for webinar Get Started with Databricks for Data Engineering session

  • 69 Views
  • 1 replies
  • 0 kudos
Latest Reply
Advika
Databricks Employee
  • 0 kudos

Hello @Meenambigai! If you have successfully enrolled in the course, open the Databricks Academy, click on the kebab menu icon (upper left corner), select "My Calendar". You’ll see the courses you’re enrolled in, organized by date. Click on the link ...

  • 0 kudos
adrjuju
by New Contributor II
  • 124 Views
  • 2 replies
  • 0 kudos

Resolved! Custom library in clean rooms

Hello Hello ! I want to use a clean room to run some algorithms developed for one of my customer without exchanging any data, the code is stored as a python library in a private git repo connected to databricks. 1 - We'd like to import the library in...

  • 124 Views
  • 2 replies
  • 0 kudos
Latest Reply
adrjuju
New Contributor II
  • 0 kudos

Thanks for the solution

  • 0 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels